Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
CRITICAL WORKER TIMEOUT when there is not enough memory , need better warning message #1646
I have a flask application which hold a article classification api(sklearn model) . And I found that one instance works well, but got
Then I visualize the memory usage and found one instance eat up 1.5G , and 5 workers 5 * 1.5 > machine's total memory . I think this cause
I think it would be better to warn user this instead of showing
And this docs http://docs.gunicorn.org/en/stable/settings.html#worker-processes
Thanks for your report.
I suspect that the memory use causes swapping or otherwise slows down operation such that a timeout does occur. Unless you can show a stack trace or some other evidence that a memory failure, like a memory allocation failure, causes this timeout, I think Gunicorn is doing everything correctly in this case.
The documentation says to "vary this a bit to find the best for your particular application’s work load." If you have some way you think the wording of that warning could be better, please feel free to open a PR.
Only by the suggestion of a collaborating friend's, I was able to come to the realization that the system was consuming ~15 GB for each worker. Which easily goes over total RAM of 32GB for 3 workers.
The setup is similar. A Flask application for a Tensorflow model that needs to load huge chunks of data, served with Gunicorn and Nginx on Ubuntu 16.04.
An explanation such as
Hope you can figure this out,
@freedrone Gunicorn has no way to know the system is out of memory, so returning such error is not possible.
To prevent the workers to use too much of memory you could limit the number of requests accepted by a worker. Also probably design your application in a way you can distribute it to not overload one machine.
@benoitc thanks for informing
ML model will consume that much RAM, and there is little I can do about it if I want it to work properly. The best I could do was to make it run on a daemonic thread so I would not get TIMEOUT because of a script that is taking too long. Rest is future work though, it is not for public eye as is. But that is a story for different time.
At least having this issue on the web helps. It can help people who are getting started to be aware that this might also be one of the reasons for TIMEOUT message (or anything of sort).
Thanks a lot for replying!