Avoid busy waiting in ioloop when system time is skipped forward

[nosyjoe]

• when the system time changes, jumps into the future make the _schedule_next method do a busy wait.
  • on slow machines (RPi), jumps of a few months into the future can block the loop for a few minutes
    => added a check for big differences in current system time and the current value of the next scheduled timeout

On a first boot of an older RPi image, tornado sometimes starts before date&time gets updated through NTP, hence blocking the ioloop for several minutes.

[bdarnell]

Instead of an arbitrary threshold, I wonder if it would be better to do something like this:

self._next_timeout += self.callback_time_sec
if self._next_timeout < current_time:
    self._next_timeout += math.ceil((current_time - self._next_timeout) / self.callback_time_sec) * self.callback_time_sec)

If I've got the math right, this is a constant-time version of the current semantics (including the fact that next_timeout always advances by a multiple of callback_time, although I don't know how important it is to preserve this behavior).

Also, systems that experience large NTP adjustments may be better served by configuring the IOLoop to use time.monotonic() instead of time.time().

[nosyjoe]

ok, I will try your solution and report back!

[nosyjoe]

I used the code you suggested with small changes. As in the original code, the _next_timeout is increased when it is smaller or equal to the current time.

I made the code calculate the next_timeout directly by tweaking your formula a bit. The new value of next_timeout is larger than current_time and smaller or equal to current_time + callback_time_sec.