Celery autoscale not working

[anishmac99]

I read from the celery documentation that we can autoscale concurrency by using autoscale option while starting a worker.

I have added 200 task in my queue and each task takes 5seconds to finish. I started worker with autoscale of 3 min concurrency and 15 max concurrency , but its starting with only 3 processes and I am not seeing any scale up in the number of processes.

Using celery with rabbitmq.

[anonymez]

We are facing the same issue with the same configuration.

We run celery with --autoscale=1024,4 but it is just stuck at 4.

I read autoscaling has different issues and should be undocumented from 4.0. Am I right? Is there any suggest solution? Starting with an high concurrency is ok?
On the other hand I also read the autoscaling code is returning in issue #3412. Is there someone can point me to some clarification post or can explain me how autoscaling is going?

Thanks.

[bb-swapnil]

I am also getting the same issue as mentioned by @anishmac99

[ChillarAnand]

Any steps to reproduce it? I am not able to reproduce it.

celery worker -l info -A t --autoscale=8,1

I have started a process and when I queued tasks, it is scaling automatically.

[pruthvirajbb]

We have already queued more than 1000 messages in rabbitmq, Each of this process is just sleeping for 5 seconds and then we started celery using this command
celery -A app worker -l info --autoscale=10,2

Though there are 1000 functions to be called, the number of workers are still 2 and doesn't scale up

[thedrow]

Autoscaling was removed in 4.x. Sorry.

[jacobwgillespie]

@thedrow are you talking about --autoscale? It doesn't appear to be removed as far as I can tell:

celery/celery/bin/worker.py

Lines 127 to 134 in 5af199c

	.. cmdoption:: --autoscale
	Enable autoscaling by providing
	max_concurrency, min_concurrency. Example::
	--autoscale=10,3
	(always keep 3 processes, but grow to 10 if necessary)

celery/celery/worker/autoscale.py

Lines 2 to 10 in 5af199c

	"""Pool Autoscaling.
	This module implements the internal thread responsible
	for growing and shrinking the pool according to the
	current autoscale settings.
	The autoscale thread is only enabled if
	the :option:`celery worker --autoscale` option is used.
	"""

[anonymez]

@jacobwgillespie, We are all here struggling with autoscale since documentation and help are wrong and not updated.

[thedrow]

The release notes for 4.0 says it is removed so it should be.
No one maintained it for a while and it has a lot of bugs which we don't have time to resolve.

[Summer-D]

If it's obsoleted, why autoscale still shows up in the 4.1 document?

[thedrow]

It should be removed.

[champ]

@thedrow I've scanned the changelogs for 4.0.x and 4.1.x - the only reference to --autoscale I found in those releases is this one in 4.0.x:

Auto-scale didn’t always update keep-alive when scaling down.

Fix contributed by Philip Garnero.

In what exact version was this removed and in what changelog was this reported?

Obviously, the references in the documentation should be removed if it's not still an option.

[illagrenan]

I think this issue should be re-opened.

A) Autoscaling was really removed --> I wasn't able to find any info about this in changelogs. And I can see Autoscaling section in the latest doc: http://docs.celeryproject.org/en/latest/userguide/workers.html#autoscaling.
B) Autoscaling wasn't removed but is probably broken.

[thedrow]

PRs are welcome to fix autoscaling or to remove it if no fixes are possible.

[warrenmcquinn]

According to the 4.2 docs, --autoscale is still an option.

[gsmethells]

@thedrow If the documentation says it is there, then I do not see how anyone could argue that it isn't supported. It is a bug until the documentation reflects that it was deprecated (properly, where usage in configuration causes a warning) and the CHANGELOG calls out its removal in explicit terms. Fix this situation before you muddy the reputation of this project. This needs to happen in version 4.x, not in version 5.x

[patrickscottbest]

I am also a little mystified on the functionality of autoscaler. Is it decided to remove based on the 2018 sprint candidate? Also, if it's still available, i'd like a scale DOWN method! If my threads are busy chewing on something, then why would i want to overburden the machine with additional workload... can this be accomplished by reversing the numbering in the --autoscale parameter ? Am I making any kind of sense here?

[dasani08]

Thank @thedrow
I have the same issue with auto-scaling until I come here. Are there any other options for us if we actually wanna scale celery worker? or may be we have to do it by ourselves.

[thedrow]

I'm super busy with 4.3 right now.
I don't have time to look into this before Celery 5.

PRs are welcome for fixes of specific problems though.

[gsmethells]

@thedrow What I recommended is not a huge time sink but will prevent the reputation of this project form being muddied. Simply remove the documentation for the autoscaling topic and add a CHANGELOG entry to the effect of why it was removed.

[dejlek]

Autoscaling is one of the coolest Celery features. I would rather remove all that useless support for various brokers and result backends than autoscaling...

[auvipy]

Autoscaling is one of the coolest Celery features. I would rather remove all that useless support for various brokers and result backends than autoscaling...

agree. can you investigate some bug n reasons of them of autoscale?

[dejlek]

I can't promise, but I will try to spend some time looking at the code... Problem is - I do not know Celery internals enough so will probably need considerate amount of time in order to be able to understand the code...

[asfaltboy]

@thedrow could there have been some confusion with the removed auto-reload feature?

The --autoreload feature has been removed.

I think that autoscaling should still work as described, as related tests are passing and no one has quoted any errors.

To everyone here that has issues, can you confirm that you are using the default Autoscaler class? And if you are, how many reserved tasks do you allow per worker (a.k.a the worker_prefetch_multiplier setting).

I'm making an assumption that many people will follow the Optimisation guide above, and set the setting to 1 (I do that as well); this will limit the maximum possible scaling to the number of worker processes currently running. In order for scaling to grow beyond that, I assume prefetch should be at least 2, to allow scaling beyond current worker count.

Alternatively, an alternative sub-class of Autoscaler can be used to debug _maybe_scale and qty methods are called and that the expected values are correctly evaluated.

Ref:

celery/celery/worker/autoscale.py

Lines 159 to 163 in 955ddf9

	@property
	def qty(self):
	return len(state.reserved_requests)

[auvipy]

can anyone help to improve the tests of auto scale

[diegopedroso1]

@anishmac99 did you scale your workers with the celery --autoscale?

[auvipy]

#6069 hey guys can you check this patch?