Number of opened sockets grow unlimited

[ThomasJannaud]

Hi

I have a very small number of requests incoming on my server and yet the flask server always ends up getting stuck/hanging indefinitely.

I tracked down the problem to : the allowed number of files that the server opens reaches ulimit.

The flask server creates sockets one after the other and I don't know why it doesn't close them. I am not doing anything crazy, just rendering a page or putting a number in a mongo database

Any thoughts ?

See full issue here : http://stackoverflow.com/questions/20540129/ever-increasing-number-of-opened-sockets-on-python-web-server

(I don't know how issues work on github, first timer here ; I don't mind copying the full issue here if you prefer)
Thanks

[ThiefMaster]

You should not use the dev server for anything but development/testing.

[ThomasJannaud]

Where do you see that I am using the dev server ? I don't think I am using it

[untitaker]

If you use app.run, you are using it. app.run uses the primitive builtin server from Werkzeug, which "works", but only well enough for testing purposes. Its only advantages are ease of use and the lack of dependencies.

See http://flask.pocoo.org/docs/deploying/ about what to actually use on your server.

[ThomasJannaud]

I have replaced app.run by tornado and got the same behaviour. (using http://flask.pocoo.org/docs/deploying/wsgi-standalone/#tornado).

[untitaker]

That's a little bit surprising. Do you get that behavior with gunicorn too?

[coldfire-x]

we are using pymongo + flask + gunicorn, and everything is ok. pymongo will maintain a connection pool, every MongoClient instance has a built-in connection pool. So, if you have too much running mongoclient instances, it may complain about too many open files.

how-does-connection-pooling-work-in-pymongo

[ThomasJannaud]

Hi
The culprit was indeed pymongo and the way I opened the client and not closing it. It seems that pymongo does the optimization on its side, so no need to try to do it in our code it will just open too many sockets
Thanks all for your help