workers unexpectedly time out if system time changes

[flixr]

Worker timeouts currently use the system clock for checking timeouts, delays, etc..
This is not robust as the system time might jump (e.g. due to NTP, PTP or other time updates)

IMHO these timeouts should use the monotonic clock as here we never care about the system time, but just the real elapsed time to check whether we need to kill a worker.

[tilgovi]

Looks like time.monotonic() is only guaranteed to exist since Python 3.5, but if it's there we should, perhaps, use it for things.

[flixr]

It should be available since Python 3.3, but we could (IMHO should) use e.g. https://pypi.python.org/pypi/monotonic to also make this fix available on older Python versions.

Quickly looking through the code it doesn't seem to be too hard except that in

gunicorn/gunicorn/workers/workertmp.py

Line 50 in 2b839ca

return os.fstat(self._tmp.fileno()).st_ctime

we would have to actually store the monotonic time in the file (or something like that) instead of using ctime of the file itself.

[duffsterlp]

We use gunicorn on an embedded system where the clock ends up jumping ahead due to particular events. When the clock jump occurs, we see this same issue.

[tilgovi]

Closed by #1967.