Replace select() syscall with Selectors module to fix filedescriptors out of range issue

[Eric-Arellano]

Problem

Twitter has 85 targets that fail to compile due to the message: filedescriptor out of range in select(), which sometimes traces to nailgun_executor.py and sometimes to util/socket.py:

pants/src/python/pants/java/nailgun_executor.py

Line 221 in 640400c

readable, _, _ = select.select([ng_stdout], [], [], (-1 * remaining_time))

pants/src/python/pants/util/socket.py

Line 67 in 640400c

readable, _, _ = safe_select([self._socket], [], [], self._select_timeout)

We don't encounter this in Pants CI, because we don't try to compile as many targets as Twitter does (1500+ for these tests).

In the process of researching this, I found that many view the select() syscall as deprecated in favor of the more modern poll() and even more modern epoll(), devpoll(), and kqueue(). See https://beesbuzz.biz/code/5739-The-problem-with-select-vs-poll and https://stackoverflow.com/questions/970979/what-are-the-differences-between-poll-and-select.

Beyond generally being less desirable to use select(), it is very likely this is causing the filedescriptor out of range issue, per https://stackoverflow.com/a/41497735.

Proposed solution: Selectors library

Python 3.4 introduced the new selectors stdlib module via PEP 3156, built right on top of the low-level select stdlib module.

Beyond nice abstractions, the main benefit we get is the class DefaultSelector, which will choose the best syscall possible for that platform, e.g. kqueue on macOS / BSD and epoll() on Ubuntu. It will only revert to select() as a last resort, which will allow us to work around the filedescriptor issue, in addition to making the code more robust and easier to understand.

Handling Python 2 compatibility

This module was only introduced in Python 3.4, and was not backported. There does exist a 3rdparty backport selectors2 that we could use.

However, because we plan to drop Python 2 any day now—and the Twitter tests were passing when using Python 2 to run Pants—I propose that we keep the Python 2 code exactly how it is using select.select(), and only use selectors for Python 3 via an if PY3 check. In hopefully a week, we can kill off the Python 2 code branch.

[blorente]

This is outstanding research, thanks!

I couldn't find the specific place where we set a high ulimit in Twitter code, but I don't think it matters because it's fairly safe to assume we do, and this change is good regardless.

I'm definitely on board with using the selectors stdlib module. For the PY2 compat, I lean in favour of keeping the old code behind a PY2 flag, I think it will make it easier to remove once we no longer support PY2, and there's very little that can go wrong.