gaierror with socket.SOCK_CLOEXEC

[behackett]

• gevent version: 1.2.1
• Python version: PyPy 5.6 (specifically the binary from pypy.org)
• Operating System: Ubuntu 12.04

Description:

The following repro script demonstrates the problem.

from gevent import monkey
monkey.patch_all()

import socket
import traceback

print(getattr(socket, 'SOCK_CLOEXEC', 0))

for res in socket.getaddrinfo('python.org', 80, socket.AF_UNSPEC, socket.SOCK_STREAM):
    af, socktype, proto, dummy, sa = res
    try:
        sock = socket.socket(af, socktype | getattr(socket, 'SOCK_CLOEXEC', 0), proto)
        sock.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)
        sock.connect(sa)
        break
    except socket.error:
        traceback.print_exc()

The problem only happens when SOCK_CLOEXEC is defined. Using the pypy portable binary avoids the issue because it doesn't define SOCK_CLOEXEC. I'm guessing this is some weird bug with the libc version in Ubuntu 12.04.

What I've run:

$ pypy-5.6-linux_x86_64-portable/bin/pypy repro.py 
0

$ pypy2-v5.6.0-linux64/bin/pypy repro.py 
524288
Traceback (most recent call last):
  File "repro.py", line 14, in <module>
    sock.connect(sa)
  File "/home/ubuntu/pypy2-v5.6.0-linux64/site-packages/gevent-1.2.1-py2.7-linux-x86_64.egg/gevent/_socket2.py", line 221, in connect
    r = getaddrinfo(address[0], address[1], sock.family, sock.type, sock.proto)
  File "/home/ubuntu/pypy2-v5.6.0-linux64/site-packages/gevent-1.2.1-py2.7-linux-x86_64.egg/gevent/_socketcommon.py", line 272, in getaddrinfo
    return get_hub().resolver.getaddrinfo(host, port, family, socktype, proto, flags)
  File "/home/ubuntu/pypy2-v5.6.0-linux64/site-packages/gevent-1.2.1-py2.7-linux-x86_64.egg/gevent/resolver_thread.py", line 65, in getaddrinfo
    return self.pool.apply(_socket.getaddrinfo, args, kwargs)
  File "/home/ubuntu/pypy2-v5.6.0-linux64/site-packages/gevent-1.2.1-py2.7-linux-x86_64.egg/gevent/pool.py", line 326, in apply
    return self.spawn(func, *args, **kwds).get()
  File "/home/ubuntu/pypy2-v5.6.0-linux64/site-packages/gevent-1.2.1-py2.7-linux-x86_64.egg/gevent/event.py", line 385, in get
    return self.get(block=False)
  File "/home/ubuntu/pypy2-v5.6.0-linux64/site-packages/gevent-1.2.1-py2.7-linux-x86_64.egg/gevent/event.py", line 375, in get
    return self._raise_exception()
  File "/home/ubuntu/pypy2-v5.6.0-linux64/site-packages/gevent-1.2.1-py2.7-linux-x86_64.egg/gevent/event.py", line 355, in _raise_exception
    reraise(*self.exc_info)
  File "/home/ubuntu/pypy2-v5.6.0-linux64/site-packages/gevent-1.2.1-py2.7-linux-x86_64.egg/gevent/threadpool.py", line 211, in _worker
    value = func(*args, **kwargs)
gaierror: [Errno -7] ai_socktype not supported

[behackett]

Just to clarify, the repro script executes without issue if you comment out the call to monkey.patch_all()

[jamadden]

I'm guessing it's something like that too, but I have no way to try to reproduce this (the oldest Ubuntu I have is 14.04).

Since 12.04 leaves support in a month, and since there is an easy, credible workaround (use the portable binary---which is what gevent is tested against anyway), and since this seems like a bug in a specific (old) platform, it doesn't seem to me that there's much gevent can or should try to do.

Thanks for the report, it may help others find the workaround.

[behackett]

Since I have access to Ubuntu 14.04 and the pypy binary claims to be for 12.04 and 14.04, I just tested there as well. Same issue.

This was reported to me as a bug in PyMongo, which I'm going to close won't fix. Feel free to close this as well if you don't think it's worth investigating further.

[jamadden]

Since you can produce it on 14.04 I'll take a look there. I don't know if there's going to be anything we can do about it, it strikes me as a libc or pypy issue, but maybe we can file an upstream issue.

[jamadden]

Very interesting. This turns out to be an issue on any version/implementation of Python 2 that has the SOCK_CLOEXEC flag available. The versions of CPython 2.7 that ship with Ubuntu up through at least 16.04 don't seem to have this flag, so it can't happen there.

First, if we use GEVENT_RESOLVER=block we can get a much more informative error message:

$  GEVENT_RESOLVER=block python /tmp/repro.py
Traceback (most recent call last):
  File "/tmp/repro.py", line 15, in <module>
    sock.connect(sa)
  File "//site-packages/gevent/_socket2.py", line 221, in connect
    r = getaddrinfo(address[0], address[1], sock.family, sock.type, sock.proto)
  File "//site-packages/gevent/_socketcommon.py", line 272, in getaddrinfo
    return get_hub().resolver.getaddrinfo(host, port, family, socktype, proto, flags)
gaierror: [Errno -7] ai_socktype not supported

(That traceback is supposed to be getting martialed across the threadpool that's used by default, so we shouldn't have to do this. I don't know why it isn't ☹️ )

We can see that the connect method needs to resolve the given address into an IP address using getaddrinfo. It passes in the family, type, and protocol of the socket to get back an address that matches what the socket uses. And SOCK_STREAM | SOCK_CLOEXEC isn't a valid type of address to request.

At this point I tested CPython 3.5, which does have the SOCK_CLOEXEC flag available, but didn't cause the error. That's because it doesn't pass the extra parameters to getaddrinfo (and if you do, you can cause the error):

r = getaddrinfo(address[0], address[1], self.family)

That doesn't seem right, though, shouldn't we need the type and protocol to be passed? So I went hunting through the CPython source code to determine how it dealt with those values.

After a maze of twisty passages all alike (CPython goes to a great deal of trouble to avoid calling getaddrinfo in special cases), I found how it was calling getaddrinfo in a connect call. From Python 3 (master):

    /* perform a name resolution */
    memset(&hints, 0, sizeof(hints));
    hints.ai_family = af;
    Py_BEGIN_ALLOW_THREADS
    ACQUIRE_GETADDRINFO_LOCK
    error = getaddrinfo(name, NULL, &hints, &res);

In other words, it's not passing the type and protocol. Python 2 does the same thing

So gevent's Python 2 connect method is being too aggressive in passing type and protocol to getaddrinfo and needs to stop.

Thanks for the report!

[behackett]

Well done!