-
Notifications
You must be signed in to change notification settings - Fork 13.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Celery autoscaling overrides normal worker_concurrency setting #8480
Comments
Thanks for opening your first issue here! Be sure to follow the issue template! |
I just ran into this too. Airflow 1.10.9 / Celery 4.4.0. This may be unrelated, but I also noticed that the worker wasn't actually autoscaling: |
Autoscaling isn't supposed to be supported in Celery 4.x, so I wouldn't be surprised to see it behaving erratically anyways. I just was surprised to see it always defaulting to 12 workers even with "-c 1" 🤔 |
Duplicate of #8044 @jsmodic you are right, autoscaling is weird in Celery in 4.x and there are issues reporting this problem in Celery itself. If you want to perform production-grade autoscaling and you are using k8s cluster this may be interesting to you: https://www.astronomer.io/blog/the-keda-autoscaler/ Indeed
@jsmodic if you think we can close the issue - please do :) |
@dimberman @ashb just an idea: maybe we should remove / deprecated the "celery autoscaling" option? |
Isn't there no worker_autoscale option set by default:
So where is this default coming from? |
I will take a closer look, but as reported on 1.10.9 changing concurrency of worker from cli level doesn't work so it's possible that the autoscaling option is also corrupted. |
@turbaszek what celery version are you using? On 1.10.9 it starts up like this for me on Celery 4.3.0 with no autoscaling configured anywhere. From Celery's stdout on startup:
On the machine:
There's only 4 cores on that machine, that's what triggered me digging into how there could possibly be 12 workers. |
I am testing this in breeze, and I've got
Of course, you can have more processes than cores. But it doesn't mean that everything will be executed at the same time :) |
Well there's this... airflow/airflow/config_templates/config.yml Lines 1114 to 1125 in 6db66ea
...but from what I can tell that file is only used for documentation, right? I can't find anything about it in the Celery source either 🤔 |
Not exactly, this file is used to generate |
My airflow.cfg (and celery_config.py) doesn't reference autoscaling at all (since I haven't updated them really since 1.10.2 or so, before this feature landed in airflow). It's interesting you can't replicate the behavior though. When I test it with my config with and without 'autoscale' in the dictionary, it will show both the bad and then expected behavior. (I'm not doing a very scientific test since I can't easily setup a clean environment, I'm editing the dist-package directly on a development machine) |
@jsmodic I was able to verify that |
The autoscale config is commented out since 1.10.10 which is a change from 1.10.9 where it isn't and I think this accounts for the difference in the behavior that people are seeing. Based on the comment above the setting, it makes sense that you wouldn't be able to set worker concurrency in 1.10.9: |
As far as I tested it should be enough to remove 'autoscale' from the dictionary to be unpacked into Celery if autoscaling isn't configured in the airflow.cfg. But I haven't confirmed that works on other versions of Celery or how it behaves if you actually desire autoscaling. |
@jsmodic from what I'm understanding you're editing this in the default configuration in airflow. From my understanding of it, I think without actually editing the package itself you are going to have trouble avoiding autoscale in 1.10.9 because it is set as a default. Given it is commented out in the package default config |
Following up on this, it's not ideal, but if in 1.10.9 you want to set a specific worker process count to, for example, 8 workers then just include: As @eeshugerman pointed out, auto-scaling is not working so even if you set the upper bound to be something higher you would still sit at 8 workers. A caveat to this, I dug into auto-scaling not working and I think autoscaling is only broken in celery with the combination of settings: |
@turbaszek I think this can be closed, correct? |
Airflow 1.10.9
Celery 4.3.0
Setting Celery 'worker_concurrency' is overriden by the autoscaling no matter what you configure.
Problematic code is here:
airflow/airflow/cli/commands/celery_command.py
Line 121 in 6db66ea
(this is head but it's the same bug in the 1.10.x version of this file)
You always get the default autoscaling setting with head if you don't set autoscaling at all but do set worker concurrency (which is 12 workers).
Not adding "autoscale" to the dictionary when autoscaling is not set is enough to solve this (at least, I tested the worker_concurrency half of things, and the asked for concurrency is back with that change).
The text was updated successfully, but these errors were encountered: