About gunicorn [CRITICAL] Worker Timeout

[skytoup]

Recently, I found that my worker appears timeout at about 9 o 'clock every day.

Finally, one day I found that when the system time was modified (more than the current time), the worker unexpectedly appeared timeout! At this time, I can confirm that the daily worker timeout is caused by time synchronization.

I checked the relevant source code and found that the system time was used to determine whether the worker was timeout.

gunicorn/gunicorn/arbiter.py

Line 497 in 7af6f65

if time.time() - worker.tmp.last_update() <= self.timeout:

[tilgovi]

This probably cannot be fixed right now. The workers chmod a temporary file to mark liveness so it's not possible to use a monotonic time there.

The best workaround is probably to use a time daemon that handles smoothing the system clock changes.

[benoitc]

hrm wouldn’t fic the file myime to a monotonic timestamp work?

On Wed 23 Jan 2019 at 07:15, Randall Leeds ***@***.***> wrote: This probably cannot be fixed right now. The workers chmod a temporary file to mark liveness so it's not possible to use a monotonic time there. The best workaround is probably to use a time daemon that handles smoothing the system clock changes. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#1965 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAA4osEE-2PpUbFKeGGD5mZpVW_2Wu6qks5vF_37gaJpZM4aODxy> .

-- Sent from my Mobile

[tilgovi]

Maybe it would work. I am reading now that os.utime supports a file descriptor argument for Python 3.3+, so we could use it on the worker temp files.

https://docs.python.org/3/library/os.html#os.utime

We could set the time using time.monotonic and I think it might work!

[tilgovi]

@skytoup do you want to try to make a patch?

[skytoup]

@skytoup do you want to try to make a patch?
--
Refer to your suggestion and make this patch(#1967).