ValueError: filedescriptor out of range in select() in urllib3

[SaveTheRbtz]

We are using requests library in our code and after recent urllib3 upgrade we've started seeing following errors in logs on our testing servers:

"/usr/lib/python2.7/site-packages/requests-2.6.0-py2.7.egg/requests/packages/urllib3/connectionpool.py", line 762, in _validate_conn
conn.connect()
File "/usr/lib/python2.7/site-packages/requests-2.6.0-py2.7.egg/requests/packages/urllib3/connection.py", line 238, in connect
ssl_version=resolved_ssl_version)
File "/usr/lib/python2.7/site-packages/requests-2.6.0-py2.7.egg/requests/packages/urllib3/contrib/pyopenssl.py", line 302, in ssl_wrap_socket
select.select([sock], [], [])
ValueError: filedescriptor out of range in select()

What it basically means is that sock that is passed to select[1] is >= 1024. That may happen for example if application opens 1024 files on start and then gets OpenSSL.SSL.WantReadError during TLS handshake while using urllib3.

Regression was introduced in: a49bec5 [pyopenssl: wait for data before handshake retry]

Simple fix is to factor out functionality for poll -> select fallback from https://github.com/shazow/urllib3/blob/master/urllib3/util/connection.py#L12 and replace all select.select() calls in urllib3 with it.

[1] select(2) is bound to FD_SETSIZE fds which equals to 1024 on most UNIX systems: http://pubs.opengroup.org/onlinepubs/007908799/xsh/select.html

[mfrobben]

Any updates on this issue?

[shazow]

@t-8ch Any thoughts on this, since you did a49bec5?

[cralston0]

Any updates?

[shazow]

@sigmavirus24 @Lukasa I'm gone for a few days, could one of you take a look please? :D

[Lukasa]

Last I checked we were still waiting on #685 to get cleaned up.

[cralston0]

Comments on the pull request indicate there were problems with it passing tests and also some design discussions to be had. I can look at the tests part, at least.

[cralston0]

pyca/pyopenssl#168

In the meantime, a workaround that seems to have addressed this for me is removing the timeout= parameter when using requests. (I realize that's one package up and one package down from here and not relevant to fixing the issue)

[jared-mackey]

Any updates?

[sethmlarson]

On non-Windows platforms (or any platform with a better selector than select.select) this will be fixed with #1001