Celery autoscale not working

[anishmac99]

I read from the celery documentation that we can autoscale concurrency by using autoscale option while starting a worker.

I have added 200 task in my queue and each task takes 5seconds to finish. I started worker with autoscale of 3 min concurrency and 15 max concurrency , but its starting with only 3 processes and I am not seeing any scale up in the number of processes.

Using celery with rabbitmq.

[anonymez]

We are facing the same issue with the same configuration.

We run celery with --autoscale=1024,4 but it is just stuck at 4.

I read autoscaling has different issues and should be undocumented from 4.0. Am I right? Is there any suggest solution? Starting with an high concurrency is ok?
On the other hand I also read the autoscaling code is returning in issue #3412. Is there someone can point me to some clarification post or can explain me how autoscaling is going?

Thanks.

[bb-swapnil]

I am also getting the same issue as mentioned by @anishmac99

[ChillarAnand]

Any steps to reproduce it? I am not able to reproduce it.

celery worker -l info -A t --autoscale=8,1

I have started a process and when I queued tasks, it is scaling automatically.

[pruthvirajbb]

We have already queued more than 1000 messages in rabbitmq, Each of this process is just sleeping for 5 seconds and then we started celery using this command
celery -A app worker -l info --autoscale=10,2

Though there are 1000 functions to be called, the number of workers are still 2 and doesn't scale up

[thedrow]

Autoscaling was removed in 4.x. Sorry.

[jacobwgillespie]

@thedrow are you talking about --autoscale? It doesn't appear to be removed as far as I can tell:

celery/celery/bin/worker.py

Lines 127 to 134 in 5af199c

	.. cmdoption:: --autoscale
	Enable autoscaling by providing
	max_concurrency, min_concurrency. Example::
	--autoscale=10,3
	(always keep 3 processes, but grow to 10 if necessary)

celery/celery/worker/autoscale.py

Lines 2 to 10 in 5af199c

	"""Pool Autoscaling.
	This module implements the internal thread responsible
	for growing and shrinking the pool according to the
	current autoscale settings.
	The autoscale thread is only enabled if
	the :option:`celery worker --autoscale` option is used.
	"""

[anonymez]

@jacobwgillespie, We are all here struggling with autoscale since documentation and help are wrong and not updated.

[thedrow]

The release notes for 4.0 says it is removed so it should be.
No one maintained it for a while and it has a lot of bugs which we don't have time to resolve.

[Summer-D]

If it's obsoleted, why autoscale still shows up in the 4.1 document?

[thedrow]

It should be removed.

[champ]

@thedrow I've scanned the changelogs for 4.0.x and 4.1.x - the only reference to --autoscale I found in those releases is this one in 4.0.x:

Auto-scale didn’t always update keep-alive when scaling down.

Fix contributed by Philip Garnero.

In what exact version was this removed and in what changelog was this reported?

Obviously, the references in the documentation should be removed if it's not still an option.

[illagrenan]

I think this issue should be re-opened.

A) Autoscaling was really removed --> I wasn't able to find any info about this in changelogs. And I can see Autoscaling section in the latest doc: http://docs.celeryproject.org/en/latest/userguide/workers.html#autoscaling.
B) Autoscaling wasn't removed but is probably broken.

[thedrow]

PRs are welcome to fix autoscaling or to remove it if no fixes are possible.

[warrenmcquinn]

According to the 4.2 docs, --autoscale is still an option.

[gsmethells]

@thedrow If the documentation says it is there, then I do not see how anyone could argue that it isn't supported. It is a bug until the documentation reflects that it was deprecated (properly, where usage in configuration causes a warning) and the CHANGELOG calls out its removal in explicit terms. Fix this situation before you muddy the reputation of this project. This needs to happen in version 4.x, not in version 5.x

[patrickscottbest]

I am also a little mystified on the functionality of autoscaler. Is it decided to remove based on the 2018 sprint candidate? Also, if it's still available, i'd like a scale DOWN method! If my threads are busy chewing on something, then why would i want to overburden the machine with additional workload... can this be accomplished by reversing the numbering in the --autoscale parameter ? Am I making any kind of sense here?