New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Celery autoscale not working #4003

Open
anishmac99 opened this Issue Apr 28, 2017 · 16 comments

Comments

@anishmac99
Copy link

anishmac99 commented Apr 28, 2017

I read from the celery documentation that we can autoscale concurrency by using autoscale option while starting a worker.

I have added 200 task in my queue and each task takes 5seconds to finish. I started worker with autoscale of 3 min concurrency and 15 max concurrency , but its starting with only 3 processes and I am not seeing any scale up in the number of processes.

Using celery with rabbitmq.

@anonymez

This comment has been minimized.

Copy link

anonymez commented May 6, 2017

We are facing the same issue with the same configuration.

We run celery with --autoscale=1024,4 but it is just stuck at 4.

I read autoscaling has different issues and should be undocumented from 4.0. Am I right? Is there any suggest solution? Starting with an high concurrency is ok?
On the other hand I also read the autoscaling code is returning in issue #3412. Is there someone can point me to some clarification post or can explain me how autoscaling is going?

Thanks.

@bb-swapnil

This comment has been minimized.

Copy link

bb-swapnil commented May 15, 2017

I am also getting the same issue as mentioned by @anishmac99

@ChillarAnand

This comment has been minimized.

Copy link
Contributor

ChillarAnand commented May 23, 2017

Any steps to reproduce it? I am not able to reproduce it.

celery worker -l info -A t --autoscale=8,1

I have started a process and when I queued tasks, it is scaling automatically.

@pruthvirajbb

This comment has been minimized.

Copy link

pruthvirajbb commented May 26, 2017

We have already queued more than 1000 messages in rabbitmq, Each of this process is just sleeping for 5 seconds and then we started celery using this command
celery -A app worker -l info --autoscale=10,2

Though there are 1000 functions to be called, the number of workers are still 2 and doesn't scale up

@thedrow

This comment has been minimized.

Copy link
Member

thedrow commented Sep 26, 2017

Autoscaling was removed in 4.x. Sorry.

@thedrow thedrow closed this Sep 26, 2017

@jacobwgillespie

This comment has been minimized.

Copy link

jacobwgillespie commented Sep 26, 2017

@thedrow are you talking about --autoscale? It doesn't appear to be removed as far as I can tell:

celery/celery/bin/worker.py

Lines 127 to 134 in 5af199c

.. cmdoption:: --autoscale
Enable autoscaling by providing
max_concurrency, min_concurrency. Example::
--autoscale=10,3
(always keep 3 processes, but grow to 10 if necessary)

"""Pool Autoscaling.
This module implements the internal thread responsible
for growing and shrinking the pool according to the
current autoscale settings.
The autoscale thread is only enabled if
the :option:`celery worker --autoscale` option is used.
"""

@anonymez

This comment has been minimized.

Copy link

anonymez commented Sep 26, 2017

@jacobwgillespie, We are all here struggling with autoscale since documentation and help are wrong and not updated.

@thedrow

This comment has been minimized.

Copy link
Member

thedrow commented Oct 1, 2017

The release notes for 4.0 says it is removed so it should be.
No one maintained it for a while and it has a lot of bugs which we don't have time to resolve.

@Summer-D

This comment has been minimized.

Copy link

Summer-D commented Oct 18, 2017

If it's obsoleted, why autoscale still shows up in the 4.1 document?

@thedrow

This comment has been minimized.

Copy link
Member

thedrow commented Oct 19, 2017

It should be removed.

@champ

This comment has been minimized.

Copy link

champ commented Jan 31, 2018

@thedrow I've scanned the changelogs for 4.0.x and 4.1.x - the only reference to --autoscale I found in those releases is this one in 4.0.x:

Auto-scale didn’t always update keep-alive when scaling down.

Fix contributed by Philip Garnero.

In what exact version was this removed and in what changelog was this reported?

Obviously, the references in the documentation should be removed if it's not still an option.

@illagrenan

This comment has been minimized.

Copy link

illagrenan commented Apr 4, 2018

I think this issue should be re-opened.

A) Autoscaling was really removed --> I wasn't able to find any info about this in changelogs. And I can see Autoscaling section in the latest doc: http://docs.celeryproject.org/en/latest/userguide/workers.html#autoscaling.
B) Autoscaling wasn't removed but is probably broken.

@thedrow thedrow reopened this Apr 5, 2018

@thedrow

This comment has been minimized.

Copy link
Member

thedrow commented Apr 5, 2018

PRs are welcome to fix autoscaling or to remove it if no fixes are possible.

@auvipy auvipy added this to the v5.0.0 milestone Apr 5, 2018

@auvipy auvipy added the NEEDS HELP label May 28, 2018

@xirdneh xirdneh self-assigned this Jul 25, 2018

@warrenmcquinn

This comment has been minimized.

Copy link

warrenmcquinn commented Oct 5, 2018

According to the 4.2 docs, --autoscale is still an option.

@gsmethells

This comment has been minimized.

Copy link

gsmethells commented Jan 28, 2019

@thedrow If the documentation says it is there, then I do not see how anyone could argue that it isn't supported. It is a bug until the documentation reflects that it was deprecated (properly, where usage in configuration causes a warning) and the CHANGELOG calls out its removal in explicit terms. Fix this situation before you muddy the reputation of this project. This needs to happen in version 4.x, not in version 5.x

@patrickscottbest

This comment has been minimized.

Copy link

patrickscottbest commented Feb 15, 2019

I am also a little mystified on the functionality of autoscaler. Is it decided to remove based on the 2018 sprint candidate? Also, if it's still available, i'd like a scale DOWN method! If my threads are busy chewing on something, then why would i want to overburden the machine with additional workload... can this be accomplished by reversing the numbering in the --autoscale parameter ? Am I making any kind of sense here?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment