Memory leak in Celery worker

[U039b]

We notice an increasing consumption of memory. Restarting the Celery worker release the memory.

• Debian Strech
• Python 3.5

The task is defined here.

It appears that task state is not released after task execution. A quick diff on:

• 1st_execution_lsof.txt
• 2nd_execution_lsof.txt

shows duplicated unreleased resources.

[JulienPalard]

Maybe using https://docs.python.org/3/library/tracemalloc.html may help here, to compare allocations of a short-lived process and a long lived one.

[U039b]

I got rid of all Popen but Celery still creates extra processes: c58c3c2

This issue is induced by Minio:

• fput_object on large file creates 3 unterminated processes minio/minio-py#602
• python minio client may cause thread exhaust minio/minio-py#595