Join GitHub today
GitHub is home to over 31 million developers working together to host and review code, manage projects, and build software together.
Sign upCelery autoscale not working #4003
Comments
This comment has been minimized.
This comment has been minimized.
anonymez
commented
May 6, 2017
We are facing the same issue with the same configuration. We run celery with --autoscale=1024,4 but it is just stuck at 4. I read autoscaling has different issues and should be undocumented from 4.0. Am I right? Is there any suggest solution? Starting with an high concurrency is ok? Thanks. |
This comment has been minimized.
This comment has been minimized.
bb-swapnil
commented
May 15, 2017
I am also getting the same issue as mentioned by @anishmac99 |
This comment has been minimized.
This comment has been minimized.
Any steps to reproduce it? I am not able to reproduce it.
I have started a process and when I queued tasks, it is scaling automatically. |
This comment has been minimized.
This comment has been minimized.
pruthvirajbb
commented
May 26, 2017
We have already queued more than 1000 messages in rabbitmq, Each of this process is just sleeping for 5 seconds and then we started celery using this command Though there are 1000 functions to be called, the number of workers are still 2 and doesn't scale up |
This comment has been minimized.
This comment has been minimized.
Autoscaling was removed in 4.x. Sorry. |
thedrow
closed this
Sep 26, 2017
This comment has been minimized.
This comment has been minimized.
jacobwgillespie
commented
Sep 26, 2017
@thedrow are you talking about Lines 127 to 134 in 5af199c celery/celery/worker/autoscale.py Lines 2 to 10 in 5af199c |
This comment has been minimized.
This comment has been minimized.
anonymez
commented
Sep 26, 2017
@jacobwgillespie, We are all here struggling with autoscale since documentation and help are wrong and not updated. |
This comment has been minimized.
This comment has been minimized.
The release notes for 4.0 says it is removed so it should be. |
This comment has been minimized.
This comment has been minimized.
Summer-D
commented
Oct 18, 2017
If it's obsoleted, why autoscale still shows up in the 4.1 document? |
This comment has been minimized.
This comment has been minimized.
It should be removed. |
This comment has been minimized.
This comment has been minimized.
champ
commented
Jan 31, 2018
@thedrow I've scanned the changelogs for 4.0.x and 4.1.x - the only reference to
In what exact version was this removed and in what changelog was this reported? Obviously, the references in the documentation should be removed if it's not still an option. |
This comment has been minimized.
This comment has been minimized.
illagrenan
commented
Apr 4, 2018
I think this issue should be re-opened. A) Autoscaling was really removed --> I wasn't able to find any info about this in changelogs. And I can see Autoscaling section in the latest doc: http://docs.celeryproject.org/en/latest/userguide/workers.html#autoscaling. |
thedrow
reopened this
Apr 5, 2018
This comment has been minimized.
This comment has been minimized.
PRs are welcome to fix autoscaling or to remove it if no fixes are possible. |
auvipy
added this to the v5.0.0 milestone
Apr 5, 2018
auvipy
added
the
NEEDS HELP
label
May 28, 2018
auvipy
removed
the
Issue Type: Question
label
Jul 8, 2018
xirdneh
self-assigned this
Jul 25, 2018
This comment has been minimized.
This comment has been minimized.
warrenmcquinn
commented
Oct 5, 2018
According to the 4.2 docs, |
auvipy
added
the
DCUS2018: Sprint Candidate
label
Oct 12, 2018
This comment has been minimized.
This comment has been minimized.
gsmethells
commented
Jan 28, 2019
•
@thedrow If the documentation says it is there, then I do not see how anyone could argue that it isn't supported. It is a bug until the documentation reflects that it was deprecated (properly, where usage in configuration causes a warning) and the CHANGELOG calls out its removal in explicit terms. Fix this situation before you muddy the reputation of this project. This needs to happen in version 4.x, not in version 5.x |
This comment has been minimized.
This comment has been minimized.
patrickscottbest
commented
Feb 15, 2019
I am also a little mystified on the functionality of autoscaler. Is it decided to remove based on the 2018 sprint candidate? Also, if it's still available, i'd like a scale DOWN method! If my threads are busy chewing on something, then why would i want to overburden the machine with additional workload... can this be accomplished by reversing the numbering in the --autoscale parameter ? Am I making any kind of sense here? |
anishmac99 commentedApr 28, 2017
I read from the celery documentation that we can autoscale concurrency by using autoscale option while starting a worker.
I have added 200 task in my queue and each task takes 5seconds to finish. I started worker with autoscale of 3 min concurrency and 15 max concurrency , but its starting with only 3 processes and I am not seeing any scale up in the number of processes.
Using celery with rabbitmq.