Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Celery autoscale not working #4003

Closed
anishmac99 opened this issue Apr 28, 2017 · 26 comments
Closed

Celery autoscale not working #4003

anishmac99 opened this issue Apr 28, 2017 · 26 comments

Comments

@anishmac99
Copy link

@anishmac99 anishmac99 commented Apr 28, 2017

I read from the celery documentation that we can autoscale concurrency by using autoscale option while starting a worker.

I have added 200 task in my queue and each task takes 5seconds to finish. I started worker with autoscale of 3 min concurrency and 15 max concurrency , but its starting with only 3 processes and I am not seeing any scale up in the number of processes.

Using celery with rabbitmq.

@anonymez
Copy link

@anonymez anonymez commented May 6, 2017

We are facing the same issue with the same configuration.

We run celery with --autoscale=1024,4 but it is just stuck at 4.

I read autoscaling has different issues and should be undocumented from 4.0. Am I right? Is there any suggest solution? Starting with an high concurrency is ok?
On the other hand I also read the autoscaling code is returning in issue #3412. Is there someone can point me to some clarification post or can explain me how autoscaling is going?

Thanks.

@bb-swapnil
Copy link

@bb-swapnil bb-swapnil commented May 15, 2017

I am also getting the same issue as mentioned by @anishmac99

@ChillarAnand
Copy link
Contributor

@ChillarAnand ChillarAnand commented May 23, 2017

Any steps to reproduce it? I am not able to reproduce it.

celery worker -l info -A t --autoscale=8,1

I have started a process and when I queued tasks, it is scaling automatically.

@pruthvirajbb
Copy link

@pruthvirajbb pruthvirajbb commented May 26, 2017

We have already queued more than 1000 messages in rabbitmq, Each of this process is just sleeping for 5 seconds and then we started celery using this command
celery -A app worker -l info --autoscale=10,2

Though there are 1000 functions to be called, the number of workers are still 2 and doesn't scale up

@thedrow
Copy link
Member

@thedrow thedrow commented Sep 26, 2017

Autoscaling was removed in 4.x. Sorry.

@thedrow thedrow closed this Sep 26, 2017
@jacobwgillespie
Copy link

@jacobwgillespie jacobwgillespie commented Sep 26, 2017

@thedrow are you talking about --autoscale? It doesn't appear to be removed as far as I can tell:

celery/celery/bin/worker.py

Lines 127 to 134 in 5af199c

.. cmdoption:: --autoscale
Enable autoscaling by providing
max_concurrency, min_concurrency. Example::
--autoscale=10,3
(always keep 3 processes, but grow to 10 if necessary)

"""Pool Autoscaling.
This module implements the internal thread responsible
for growing and shrinking the pool according to the
current autoscale settings.
The autoscale thread is only enabled if
the :option:`celery worker --autoscale` option is used.
"""

@anonymez
Copy link

@anonymez anonymez commented Sep 26, 2017

@jacobwgillespie, We are all here struggling with autoscale since documentation and help are wrong and not updated.

@thedrow
Copy link
Member

@thedrow thedrow commented Oct 1, 2017

The release notes for 4.0 says it is removed so it should be.
No one maintained it for a while and it has a lot of bugs which we don't have time to resolve.

@Summer-D
Copy link

@Summer-D Summer-D commented Oct 18, 2017

If it's obsoleted, why autoscale still shows up in the 4.1 document?

@thedrow
Copy link
Member

@thedrow thedrow commented Oct 19, 2017

It should be removed.

@champ
Copy link

@champ champ commented Jan 31, 2018

@thedrow I've scanned the changelogs for 4.0.x and 4.1.x - the only reference to --autoscale I found in those releases is this one in 4.0.x:

Auto-scale didn’t always update keep-alive when scaling down.

Fix contributed by Philip Garnero.

In what exact version was this removed and in what changelog was this reported?

Obviously, the references in the documentation should be removed if it's not still an option.

@illagrenan
Copy link

@illagrenan illagrenan commented Apr 4, 2018

I think this issue should be re-opened.

A) Autoscaling was really removed --> I wasn't able to find any info about this in changelogs. And I can see Autoscaling section in the latest doc: http://docs.celeryproject.org/en/latest/userguide/workers.html#autoscaling.
B) Autoscaling wasn't removed but is probably broken.

@thedrow thedrow reopened this Apr 5, 2018
@thedrow
Copy link
Member

@thedrow thedrow commented Apr 5, 2018

PRs are welcome to fix autoscaling or to remove it if no fixes are possible.

@auvipy auvipy added this to the v5.0.0 milestone Apr 5, 2018
@xirdneh xirdneh self-assigned this Jul 25, 2018
@warrenmcquinn
Copy link

@warrenmcquinn warrenmcquinn commented Oct 5, 2018

According to the 4.2 docs, --autoscale is still an option.

@gsmethells
Copy link

@gsmethells gsmethells commented Jan 28, 2019

@thedrow If the documentation says it is there, then I do not see how anyone could argue that it isn't supported. It is a bug until the documentation reflects that it was deprecated (properly, where usage in configuration causes a warning) and the CHANGELOG calls out its removal in explicit terms. Fix this situation before you muddy the reputation of this project. This needs to happen in version 4.x, not in version 5.x

@patrickscottbest
Copy link

@patrickscottbest patrickscottbest commented Feb 15, 2019

I am also a little mystified on the functionality of autoscaler. Is it decided to remove based on the 2018 sprint candidate? Also, if it's still available, i'd like a scale DOWN method! If my threads are busy chewing on something, then why would i want to overburden the machine with additional workload... can this be accomplished by reversing the numbering in the --autoscale parameter ? Am I making any kind of sense here?

@dasani08
Copy link

@dasani08 dasani08 commented Mar 13, 2019

Thank @thedrow
I have the same issue with auto-scaling until I come here. Are there any other options for us if we actually wanna scale celery worker? or may be we have to do it by ourselves.

@thedrow
Copy link
Member

@thedrow thedrow commented Mar 13, 2019

I'm super busy with 4.3 right now.
I don't have time to look into this before Celery 5.

PRs are welcome for fixes of specific problems though.

@gsmethells
Copy link

@gsmethells gsmethells commented May 10, 2019

@thedrow What I recommended is not a huge time sink but will prevent the reputation of this project form being muddied. Simply remove the documentation for the autoscaling topic and add a CHANGELOG entry to the effect of why it was removed.

@auvipy auvipy modified the milestones: v5.0.0, 4.5 May 10, 2019
@dejlek
Copy link
Contributor

@dejlek dejlek commented Jun 3, 2019

Autoscaling is one of the coolest Celery features. I would rather remove all that useless support for various brokers and result backends than autoscaling...

@auvipy
Copy link
Member

@auvipy auvipy commented Jun 7, 2019

Autoscaling is one of the coolest Celery features. I would rather remove all that useless support for various brokers and result backends than autoscaling...

agree. can you investigate some bug n reasons of them of autoscale?

@dejlek
Copy link
Contributor

@dejlek dejlek commented Jun 7, 2019

I can't promise, but I will try to spend some time looking at the code... Problem is - I do not know Celery internals enough so will probably need considerate amount of time in order to be able to understand the code...

@auvipy auvipy modified the milestones: 4.5, 4.7 Jun 8, 2019
@asfaltboy
Copy link
Contributor

@asfaltboy asfaltboy commented Jun 10, 2019

@thedrow could there have been some confusion with the removed auto-reload feature?

The --autoreload feature has been removed.

I think that autoscaling should still work as described, as related tests are passing and no one has quoted any errors.

To everyone here that has issues, can you confirm that you are using the default Autoscaler class? And if you are, how many reserved tasks do you allow per worker (a.k.a the worker_prefetch_multiplier setting).

I'm making an assumption that many people will follow the Optimisation guide above, and set the setting to 1 (I do that as well); this will limit the maximum possible scaling to the number of worker processes currently running. In order for scaling to grow beyond that, I assume prefetch should be at least 2, to allow scaling beyond current worker count.

Alternatively, an alternative sub-class of Autoscaler can be used to debug _maybe_scale and qty methods are called and that the expected values are correctly evaluated.

Ref:

@property
def qty(self):
return len(state.reserved_requests)

@auvipy
Copy link
Member

@auvipy auvipy commented Jun 11, 2019

can anyone help to improve the tests of auto scale

ivoire added a commit to ivoire/KissCache that referenced this issue Aug 6, 2019
Autoscaling and worker_prefetch_multiplier=1 does not work well together.
See celery/celery#4003
@auvipy auvipy modified the milestones: 4.7, 4.6 Aug 23, 2019
@auvipy auvipy modified the milestones: 4.6, 4.5 Dec 16, 2019
@diegopedroso1
Copy link

@diegopedroso1 diegopedroso1 commented Apr 27, 2020

@anishmac99 did you scale your workers with the celery --autoscale?

@auvipy
Copy link
Member

@auvipy auvipy commented May 4, 2020

#6069 hey guys can you check this patch?

@auvipy auvipy modified the milestones: 4.5, 4.4.x May 4, 2020
@auvipy auvipy closed this May 4, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet