PyPy + eventlet + urllib3: ValueError: file descriptor cannot be a negative integer (-1) #213

rouge8 · 2015-03-15T18:23:57Z

When using urllib3's connection pooling with PyPy 2.4/2.5 on both OS X and Linux with eventlet 0.17.1, I get occasional file descriptor errors:

$ python --version && TARGET_HOST=http://127.0.0.1:8000 python make_requests.py
Python 2.7.8 (10f1b29a2bd21f837090286174a9ca030b8680b2, Feb 05 2015, 17:48:23)
[PyPy 2.5.0 with GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.56)]
FAILED AT REQUEST 271
Traceback (most recent call last):
  File "app_main.py", line 75, in run_toplevel
  File "make_requests.py", line 14, in <module>
    pool.request('HEAD', TARGET_PATH)
  File "/Users/rouge8/.virtualenvs/pypy-bug/site-packages/urllib3/request.py", line 68, in request
    **urlopen_kw)
  File "/Users/rouge8/.virtualenvs/pypy-bug/site-packages/urllib3/request.py", line 81, in request_encode_url
    return self.urlopen(method, url, **urlopen_kw)
  File "/Users/rouge8/.virtualenvs/pypy-bug/site-packages/urllib3/connectionpool.py", line 533, in urlopen
    conn = self._get_conn(timeout=pool_timeout)
  File "/Users/rouge8/.virtualenvs/pypy-bug/site-packages/urllib3/connectionpool.py", line 237, in _get_conn
    if conn and is_connection_dropped(conn):
  File "/Users/rouge8/.virtualenvs/pypy-bug/site-packages/urllib3/util/connection.py", line 39, in is_connection_dropped
    p.register(sock, POLLIN)
ValueError: file descriptor cannot be a negative integer (-1)

I have sample code and reproduction instructions in a separate repo. I've also opened an issue in the PyPy tracker.

temoto · 2015-03-15T22:46:38Z

@rouge8 good news, I now have Mac to debug this issue. Bad news, the most I can offer this week is short talk. What immediately strikes my attention is range(10000) loop. Please, post ulimit -n and if it's less than 40k, try to increase it for both server and client processes. One way to increase nofile limit is /etc/security/limits.conf, another way is root shell:

$ whoami
user
$ ulimit -n
4864

$ sudo -i bash
(inception level 2)
# whoami
root
# ulimit -n 150100

# sudo -i user
(inception level 3)
$ test `ulimit -n` -eq 150100 && echo "nofile limit ok"
$ {run server} &
$ TARGET_HOST...

rouge8 · 2015-03-15T22:56:40Z

When I was initially encountered this with eventlet 0.15.2, the loop only needed to go to 1000 to reliably crash. Just tried Ubuntu 14.04 with a ulimit of 150100:

FAILED AT REQUEST 343
Traceback (most recent call last):
  File "app_main.py", line 75, in run_toplevel
  File "make_requests.py", line 14, in <module>
    pool.request('HEAD', TARGET_PATH)
  File "/home/andy/.virtualenvs/pypy-bug/site-packages/urllib3/request.py", line 68, in request
    **urlopen_kw)
  File "/home/andy/.virtualenvs/pypy-bug/site-packages/urllib3/request.py", line 81, in request_encode_url
    return self.urlopen(method, url, **urlopen_kw)
  File "/home/andy/.virtualenvs/pypy-bug/site-packages/urllib3/connectionpool.py", line 544, in urlopen
    body=body, headers=headers)
  File "/home/andy/.virtualenvs/pypy-bug/site-packages/urllib3/connectionpool.py", line 372, in _make_request
    httplib_response = conn.getresponse(buffering=True)
  File "/usr/lib/pypy/lib-python/2.7/httplib.py", line 1068, in getresponse
    response.begin()
  File "/usr/lib/pypy/lib-python/2.7/httplib.py", line 409, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/pypy/lib-python/2.7/httplib.py", line 365, in _read_status
    line = self.fp.readline(_MAXLINE + 1)
  File "/usr/lib/pypy/lib-python/2.7/socket.py", line 534, in readline
    data = self._sock.recv(self._rbufsize)
  File "/home/andy/.virtualenvs/pypy-bug/site-packages/eventlet/greenio/base.py", line 326, in recv
    timeout_exc=socket.timeout("timed out"))
  File "/home/andy/.virtualenvs/pypy-bug/site-packages/eventlet/greenio/base.py", line 201, in _trampoline
    mark_as_closed=self._mark_as_closed)
  File "/home/andy/.virtualenvs/pypy-bug/site-packages/eventlet/hubs/__init__.py", line 158, in trampoline
    listener = hub.add(hub.READ, fileno, current.switch, current.throw, mark_as_closed)
  File "/home/andy/.virtualenvs/pypy-bug/site-packages/eventlet/hubs/epolls.py", line 53, in add
    self.register(fileno, new=True)
  File "/home/andy/.virtualenvs/pypy-bug/site-packages/eventlet/hubs/poll.py", line 45, in register
    self.poll.register(fileno, mask)
IOError: [Errno 9] Bad file descriptor

temoto · 2015-03-15T23:40:29Z

@rouge8 thanks for quick response. Since first report was from another OS, please, also run the same test on same Ubuntu but without modifying ulimit. (still print ulimit -n value for info though)

rouge8 · 2015-03-16T03:23:39Z

ulimit -n was 1024.

$ python --version && TARGET_HOST=http://127.0.0.1:8000 python make_requests.py
Python 2.7.8 (2.5.0+dfsg-2~ppa1+ubuntu14.04, Feb 11 2015, 21:44:50)
[PyPy 2.5.0 with GCC 4.8.2]
FAILED AT REQUEST 281
Traceback (most recent call last):
  File "app_main.py", line 75, in run_toplevel
  File "make_requests.py", line 14, in <module>
    pool.request('HEAD', TARGET_PATH)
  File "/home/andy/.virtualenvs/pypy-bug/site-packages/urllib3/request.py", line 68, in request
    **urlopen_kw)
  File "/home/andy/.virtualenvs/pypy-bug/site-packages/urllib3/request.py", line 81, in request_encode_url
    return self.urlopen(method, url, **urlopen_kw)
  File "/home/andy/.virtualenvs/pypy-bug/site-packages/urllib3/connectionpool.py", line 533, in urlopen
    conn = self._get_conn(timeout=pool_timeout)
  File "/home/andy/.virtualenvs/pypy-bug/site-packages/urllib3/connectionpool.py", line 237, in _get_conn
    if conn and is_connection_dropped(conn):
  File "/home/andy/.virtualenvs/pypy-bug/site-packages/urllib3/util/connection.py", line 39, in is_connection_dropped
    p.register(sock, POLLIN)
ValueError: file descriptor cannot be a negative integer (-1)

davidszotten · 2015-04-07T13:22:26Z

only had a brief look so far, but i think armin rigo's suggestion might have some good legs: https://bitbucket.org/pypy/pypy/issue/1941/pypy-eventlet-urllib3-valueerror-file#comment-16423896

"It would be possible, for example, that two different objects have both a reference to some version of a socket object; one of the objects is forgotten; and that forgotten object has a __del__ that closes the socket. But the socket is still visible from the other object, and will thus appear to be suddenly closed at random times."

i don't recall where now, but i think i've seen other issues in the past with what looked like the same issue (multiple GreenSockets wrapping the same underlying socket, and one being garbage collected causing the other one to be closed)

don't know the internals enough to be able to tell if multiple greensockets on top of the same underlying socket means something has already gone wrong, or if the __del__ is unsafe

temoto · 2015-04-07T20:16:11Z

@rouge8 there was a PyPy double fd close bug fixed recently. Could you try to run your test again with recent master version of eventlet?

rouge8 · 2015-04-07T22:03:10Z

On OS X 10.9:

$ pip freeze -l | grep eventlet && python --version && TARGET_HOST=http://127.0.0.1:8000 python make_requests.py
-e git://github.com/eventlet/eventlet.git@390e71d89883649dd55fbb70dc8686bf99e2a2e2#egg=eventlet-master
Python 2.7.9 (9c4588d731b7fe0b08669bd732c2b676cb0a8233, Mar 31 2015, 07:51:42)
[PyPy 2.5.1 with GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.57)]
FAILED AT REQUEST 1946
Traceback (most recent call last):
  File "<builtin>/app_main.py", line 75, in run_toplevel
  File "make_requests.py", line 14, in <module>
    pool.request('HEAD', TARGET_PATH)
  File "/Users/rouge8/.virtualenvs/eventlet-bug/site-packages/urllib3/request.py", line 68, in request
    **urlopen_kw)
  File "/Users/rouge8/.virtualenvs/eventlet-bug/site-packages/urllib3/request.py", line 81, in request_encode_url
    return self.urlopen(method, url, **urlopen_kw)
  File "/Users/rouge8/.virtualenvs/eventlet-bug/site-packages/urllib3/connectionpool.py", line 533, in urlopen
    conn = self._get_conn(timeout=pool_timeout)
  File "/Users/rouge8/.virtualenvs/eventlet-bug/site-packages/urllib3/connectionpool.py", line 237, in _get_conn
    if conn and is_connection_dropped(conn):
  File "/Users/rouge8/.virtualenvs/eventlet-bug/site-packages/urllib3/util/connection.py", line 39, in is_connection_dropped
    p.register(sock, POLLIN)
ValueError: file descriptor cannot be a negative integer (-1)

It fails at the same request 1946 every time. I don't have access to the Ubuntu machine I was testing on, but I can launch a VM and run the tests there if that helps.

temoto · 2015-04-12T04:38:36Z

Thanks for testing again. Somehow it slipped through my inbox attention. Will check what I can do next week, but this issue is crying for warm contributor hands.

rouge8 · 2015-04-18T05:52:37Z

I don't know that I can track down and fix it, but if there's anything else I can do to help debug (even if time consuming), I'm open for suggestions.

rouge8 · 2015-05-19T05:33:26Z

Tested again with newer versions, still fails:

$ pip freeze -l | grep eventlet && python --version && TARGET_HOST=http://127.0.0.1:8000 python make_requests.py
eventlet==0.17.4
Python 2.7.9 (9c4588d731b7fe0b08669bd732c2b676cb0a8233, Mar 31 2015, 07:55:22)
[PyPy 2.5.1 with GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.57)]
FAILED AT REQUEST 2022

temoto added importance-bug need-contributor labels Apr 12, 2015

davidszotten mentioned this issue Apr 21, 2015

Pypy 231 support nameko/nameko#146

Closed

mattbennett mentioned this issue Apr 24, 2015

PyPy support nameko/nameko#248

Open

ysingh7 mentioned this issue Jul 7, 2016

Eventlet with pypy : Bad File Descriptor Error #318

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

PyPy + eventlet + urllib3: ValueError: file descriptor cannot be a negative integer (-1) #213

PyPy + eventlet + urllib3: ValueError: file descriptor cannot be a negative integer (-1) #213

rouge8 commented Mar 15, 2015

temoto commented Mar 15, 2015

rouge8 commented Mar 15, 2015

temoto commented Mar 15, 2015

rouge8 commented Mar 16, 2015

davidszotten commented Apr 7, 2015

temoto commented Apr 7, 2015

rouge8 commented Apr 7, 2015

temoto commented Apr 12, 2015

rouge8 commented Apr 18, 2015

rouge8 commented May 19, 2015

PyPy + eventlet + urllib3: ValueError: file descriptor cannot be a negative integer (-1) #213

PyPy + eventlet + urllib3: ValueError: file descriptor cannot be a negative integer (-1) #213

Comments

rouge8 commented Mar 15, 2015

temoto commented Mar 15, 2015

rouge8 commented Mar 15, 2015

temoto commented Mar 15, 2015

rouge8 commented Mar 16, 2015

davidszotten commented Apr 7, 2015

temoto commented Apr 7, 2015

rouge8 commented Apr 7, 2015

temoto commented Apr 12, 2015

rouge8 commented Apr 18, 2015

rouge8 commented May 19, 2015