Skip to content

Commit

Permalink
prevent celery command autoscale misconfig (#36576)
Browse files Browse the repository at this point in the history
  • Loading branch information
Bowrna committed Jan 28, 2024
1 parent 7bff7a5 commit 4fa8e45
Showing 1 changed file with 4 additions and 3 deletions.
7 changes: 4 additions & 3 deletions airflow/providers/celery/provider.yaml
Expand Up @@ -119,9 +119,10 @@ config:
default: "16"
worker_autoscale:
description: |
The maximum and minimum concurrency that will be used when starting workers with the
``airflow celery worker`` command (always keep minimum processes, but grow
to maximum if necessary). Note the value should be max_concurrency,min_concurrency
The maximum and minimum number of pool processes that will be used to dynamically resize
the pool based on load.Enable autoscaling by providing max_concurrency,min_concurrency
with the ``airflow celery worker`` command (always keep minimum processes,
but grow to maximum if necessary).
Pick these numbers based on resources on worker box and the nature of the task.
If autoscale option is available, worker_concurrency will be ignored.
https://docs.celeryq.dev/en/latest/reference/celery.bin.worker.html#cmdoption-celery-worker-autoscale
Expand Down

0 comments on commit 4fa8e45

Please sign in to comment.