worker-short-running is a memory hog

[jpopelka]

After bumping our base image to F37, the worker-short-running leaks memory and is periodically OOMKilled and restarted.

The worker-long-running is OK so it's just the multi-threaded one.

We can try the other thread pool, i.e. eventlet instead of gevent.
We can also experiment with setting some options, but they seem to be supported only by the prefork pool.

Also read through celery/celery#4843 (EDIT: that seems to apply only to prefork pool and RabbitMQ broker)

[TomasTomecek]

thank you for creating this, was about to do the same :D

I stumbled upon Memory leak in run Celery Django, Gevent strategy with Multi Worker but it doesn't seem related: I couldn't find any warning in the celery startup.

[jpopelka]

I tried to give the worker a 1GiB mem limit to see if it just doesn't need more memory now and it ate all of it until oomkilled

[jpopelka]

I tried the eventlet pool (#1825) end there were some PostgreSQL issues so that's not a way.

F37 contains celery-5.3.0b1, while F35 had celery-5.2.6 so I built a worker image with 5.2.6 from PyPI (rpm can't be installed because of dependencies) and I'm running it on stg now.