Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Too many open files #239

Closed
daleharvey opened this issue Nov 5, 2011 · 81 comments
Closed

Too many open files #239

daleharvey opened this issue Nov 5, 2011 · 81 comments

Comments

@daleharvey
Copy link

@daleharvey daleharvey commented Nov 5, 2011

I am building a basic load generator and started running into file descriptor limits, I havent seen any documentation pertaining to how to release resources, so either I am doing it wrong and the docs need updated, or requests is leaking file descriptors somewhere (without support for keepalive I am slightly confused at why any files would be left open at all)

@kennethreitz
Copy link
Contributor

@kennethreitz kennethreitz commented Nov 5, 2011

Where you using requests.async?

@daleharvey
Copy link
Author

@daleharvey daleharvey commented Nov 5, 2011

nope, all requests were reasonably plain requests.get / requests.post, I am still seeing a few in there

$ lsof | grep localhost | wc -l
110

all but 4/5 of them are of the format

Python    82117 daleharvey  123u    IPv4 0xffffff800da304e0       0t0      TCP localhost:61488->localhost:http (CLOSE_WAIT)

@kennethreitz
Copy link
Contributor

@kennethreitz kennethreitz commented Nov 26, 2011

I'm a bit baffled by this, to be honest.

@daleharvey
Copy link
Author

@daleharvey daleharvey commented Nov 26, 2011

Hah ill take another shot at reproducing it reliably, if I cant ill close

@kennethreitz
Copy link
Contributor

@kennethreitz kennethreitz commented Nov 26, 2011

I've seen this happening to me, but only when I'm using the async module w/ 200+ simultaneous connections.

@mike-plivo
Copy link

@mike-plivo mike-plivo commented Dec 13, 2011

Hi,
I got exactly the same problem using requests and monkey patching with gevent : some connections staying in CLOSE_WAIT .
Maybe a problem with gevent so .

@juanriaza
Copy link
Contributor

@juanriaza juanriaza commented Dec 22, 2011

It may be problem of ulimit -n. Try with a higher value.

@mike-plivo
Copy link

@mike-plivo mike-plivo commented Dec 22, 2011

"Too many open files" is the result of the bug caused by sockets staying in CLOSE_WAIT .
So ulimit won't fix just make a workaround .

@kennethreitz
Copy link
Contributor

@kennethreitz kennethreitz commented Dec 22, 2011

@Tamiel how do we fix this?

@mike-plivo
Copy link

@mike-plivo mike-plivo commented Dec 22, 2011

I will do more tests asap and try to fix .

@kennethreitz
Copy link
Contributor

@kennethreitz kennethreitz commented Dec 22, 2011

I've looked into it, and seems to be a problem with all libraries using httplib.HTTPSConnection.

@mike-plivo
Copy link

@mike-plivo mike-plivo commented Dec 22, 2011

Posted an example here :

https://gist.github.com/1512329

@acdha
Copy link
Contributor

@acdha acdha commented Jan 17, 2012

I just encountered a very similar error using an async pool with only HTTP connections - I'm still investigating but passing a pool size to async.map makes the error reproduce quickly.

@bevenky
Copy link

@bevenky bevenky commented Jan 29, 2012

Any fixes to this? This makes Requests unusable with gevent..

@kennethreitz
Copy link
Contributor

@kennethreitz kennethreitz commented Jan 29, 2012

It's all about the CLOSE_WAITs. Just have to close them. I'm not sure why they're still open though.

@bevenky
Copy link

@bevenky bevenky commented Jan 29, 2012

Is it a urllib3 issue? Having to close these by ourselves isnt a great idea i feel.

@kennethreitz
Copy link
Contributor

@kennethreitz kennethreitz commented Jan 29, 2012

It's more of a general issue. We can keep the conversation here.

@bevenky
Copy link

@bevenky bevenky commented Jan 29, 2012

Ok just to give you a perspective, we are trying to move from httplib2 to requests, and we dont see this issue with httplib2. So its not a general issue for sure.

@kennethreitz
Copy link
Contributor

@kennethreitz kennethreitz commented Jan 29, 2012

By general i mean that it's a very serious issue that effects everyone involved.

@bevenky
Copy link

@bevenky bevenky commented Jan 29, 2012

so how do we solve this? we really want to use requests + slumber moving forward

@kennethreitz
Copy link
Contributor

@kennethreitz kennethreitz commented Jan 29, 2012

I'd love to know the answer to that.

@acdha
Copy link
Contributor

@acdha acdha commented Jan 29, 2012

The leak appears to be due to the internal redirect handling, which causes new requests to be generated before the pending responses have been consumed. In testing acdha/requests@730c0e2 has an under-satisfying but effective fix simply by forcing each response to be consumed before continuing.

This required changes in two places which makes me want to refactor the interface slightly but I'm out of time to continue currently.

kennethreitz pushed a commit that referenced this issue Jan 29, 2012
@acdha
Copy link
Contributor

@acdha acdha commented Jan 30, 2012

#399 has a fix which works well in my async load generator (https://github.com/acdha/webtoolbox/blob/master/bin/http_bench.py) with thousands of requests and a low fd ulimit

@bborowin
Copy link

@bborowin bborowin commented Apr 27, 2012

I have run into the same issue when using async -- kludged a workaround by chunking requests and deleting responses / calling gc.collect

@Cabalist
Copy link

@Cabalist Cabalist commented Apr 30, 2012

I believe I was running into this today connecting to a licensed server that only allows 5 connections.

Using async I could only GET 4 things before it paused for 60 seconds.

Using the normal GET with consumption I could fetch about 150 things serially in under 40 seconds.

Haven't made my kludge yet since I saw this issue.

@dalanmiller
Copy link
Contributor

@dalanmiller dalanmiller commented Apr 30, 2012

Just got this error while using ipython and got this message. This is just making each request one at a time, but I think I got something similar when using async.

ERROR: Internal Python error in the inspect module.
Below is the traceback from this internal error.
Traceback (most recent call last):
    File "/Library/Python/2.7/site-packages/IPython/core/ultratb.py", line 756, in structured_traceback
    File "/Library/Python/2.7/site-packages/IPython/core/ultratb.py", line 242, in _fixed_getinnerframes
    File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 1035, in getinnerframes
    File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 995, in getframeinfo
    File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 456, in getsourcefile
    File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 485, in getmodule
    File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 469, in getabsfile
    File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/posixpath.py", line 347, in abspath
OSError: [Errno 24] Too many open files

Unfortunately, your original traceback can not be constructed.

Oddly, I think when using just the normal Python interpreter I get a "Max Retries Error" but I think that is another issue with me doing requests on all the same domain, but not sure.

sigmavirus24 referenced this issue in sigmavirus24/urllib3 Dec 4, 2013
- The comment in the method explains why we need to check self.pool first
@polvoazul
Copy link

@polvoazul polvoazul commented Nov 25, 2014

Should we re-open this?
I am having the same issue!

@Lukasa
Copy link
Member

@Lukasa Lukasa commented Nov 25, 2014

@polvoazul There's no way this is the same issue, which was originally reported in 2011, so I don't think reopening is correct. However, if you're running the current release of requests (2.4.3) and can reproduce the problem, opening a new issue would be correct.

@mygoda
Copy link

@mygoda mygoda commented Jan 23, 2016

@Lukasa i need you help 。 i use eventlet + requests,that always create so many sock that can't identify protocol 。 my requests is 2.4.3, is eventlet + requests cause this problem?

@Lukasa
Copy link
Member

@Lukasa Lukasa commented Jan 23, 2016

I'm sorry @mygoda, but it's impossible to know. If you aren't constraining the number of requests that can be outstanding at any one time then it's certainly possible, but that's an architectural problem outside the remit of requests.

@mygoda
Copy link

@mygoda mygoda commented Jan 23, 2016

@Lukasa thank you。 i think my issue is similar with this。 my project is pyvmomi. that connection is long-connection. i always confused why can hold so many can't identify protocol sock

@1a1a11a
Copy link

@1a1a11a 1a1a11a commented Mar 17, 2016

Having the same problem now, running 120 threads, cause 100000+ opened files, any solution right now?

@kennethreitz
Copy link
Contributor

@kennethreitz kennethreitz commented Mar 17, 2016

@mygoda you use awesome periods。

@Lukasa
Copy link
Member

@Lukasa Lukasa commented Mar 17, 2016

@1a1a11a What files do you have open? That'd be a useful first step to understanding this problem.

@sigmavirus24
Copy link
Contributor

@sigmavirus24 sigmavirus24 commented Mar 17, 2016

@1a1a11a what version of requests are you using? What version of python? What operating system? Can we get any information?

@1a1a11a
Copy link

@1a1a11a 1a1a11a commented Mar 17, 2016

I am using request 2.9.1, python 3.4, ubuntu 14.04, basically I am writing a crawler using 30 threads with proxies to crawl some website. Currently I have adjusted the file limit per process to 655350, otherwise it will report error.

@miloharper
Copy link

@miloharper miloharper commented Mar 17, 2017

I am still receiving the error "Failed to establish a new connection: [Errno 24] Too many open files" from requests.packages.urllib3.connection.VerifiedHTTPSConnection." I'm using Python 3.4, requests 2.11.1 and requests-futures 0.9.7. I appreciate requests-futures is a separate library, but it seems like the error is coming from requests. I'm attempting to make 180k asynchronous requests over SSL. I've divided those requests into segments of 1000, so I only move onto the next 1000 once all the future objects have been resolved. I'm running Ubuntu 16.04.2 and my default open files limit is 1024. It would be good to understand the underlying reason for this error. Does the requests library create an open file for each individual request? And if so, why? Is this a SSL certificate file? And does the requests library automatically close those open files when the future object is resolved?

@Lukasa
Copy link
Member

@Lukasa Lukasa commented Mar 17, 2017

Requests opens many files. Some of those files are opened for certificates, but they are opened by OpenSSL and not by Requests, so those aren't maintained. Additionally, Requests will also open, if needed, the .netrc file, the hosts file, and many others.

You will be best served by using a tool like strace to work out which files are opened. There is a strict list of system calls that lead to file descriptors being allocated, so you should reasonably swiftly be able to enumerate them. That will also let you know whether there is a problem or not. But, yes, I'd expect that if you're actively making 1000 connections over HTTPS then at peak load we could easily use over 1000 FDs.

@tkaria
Copy link

@tkaria tkaria commented Jul 11, 2017

I struggled with this issue as well and found that using opensnoop on OS X worked great to let me see what was happening if anyone runs in to the same issues.

@nyurik
Copy link

@nyurik nyurik commented Aug 29, 2017

I'm also frequently seeing this error when repeatedly calling requests.post(url, data=data) to an HTTP (not HTTPS) server. Running on Ubuntu 16.04.3, Python 3.5.2, requests 2.9.1

@Lukasa
Copy link
Member

@Lukasa Lukasa commented Aug 29, 2017

What is data?

@nyurik
Copy link

@nyurik nyurik commented Aug 29, 2017

A few hundred kb text

@Lukasa
Copy link
Member

@Lukasa Lukasa commented Aug 29, 2017

Not a file object?

@nyurik
Copy link

@nyurik nyurik commented Aug 29, 2017

No, I form a large query in memory.

@Lukasa
Copy link
Member

@Lukasa Lukasa commented Aug 29, 2017

Are you running this code in multiple threads?

@nyurik
Copy link

@nyurik nyurik commented Aug 29, 2017

No, single thread, posting to localhost

@Lukasa
Copy link
Member

@Lukasa Lukasa commented Aug 29, 2017

It seems almost impossible for us to be leaking that many FDs then: we should be repeatedly using the same TCP connection or aggressively closing it. Want to check what your server is up to?

@iraykhel
Copy link

@iraykhel iraykhel commented Sep 26, 2017

I am having this problem. Python 2.7, requests 2.18.4, urllib3 1.22.
Running multi-threaded code (not multi-processed). Connecting to at most 6 URLs at one time, manually creating and closing a new session for each one.

@mcobzarenco
Copy link

@mcobzarenco mcobzarenco commented Jun 26, 2018

I'm having the same problem on Python 3.5, requests==2.18.4

@arryon
Copy link

@arryon arryon commented Aug 8, 2018

@mcobzarenco are you sure you are (implicitly) closing the underlying connection of the response? Simply returning the response will not close the connection. When reading response.content the data is actually read and after that the socket will not stay in CLOSE_WAIT.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet