Documentation Change: for --timeout and --graceful-timeout

[katzlbt]

Suggestion for documentation of https://docs.gunicorn.org/en/stable/settings.html#timeout

Change doc --timeout:
OLD: Workers silent for more than this many seconds are killed and restarted. (what means "silent"?)
NEW: Sync-workers taking longer than --timeout to respond to a single request will receive by a catchable SIGTERM resulting in a SysExit exception. This cannot be used to save work as immediately afterwards an uncatchable SIGKILL is sent to the worker. Gthread-workers may process a single request indefinitely, even if --timeout is set.

Change doc --graceful-timeout:
OLD: Timeout for graceful workers restart.
NEW: Set a timeout before workers get an uncatchable SIGKILL when gunicorn receives SIGHUP to perform a graceful shutdown.

Thought: Why would a graceful shutdown need a different timeout, than the standard? If the server closes new connections they must finish after --timeout or get killed anyway.

#2839

[tilgovi]

Timeout is not just for synchronous workers, though it functions as a request timeout only for those.

I've thought for a while that adding a --request-timeout option might be the most clear and backwards compatible thing, but would require implementing that for all workers.

"Silent" means not touching a heartbeat file. Asynchronous workers can also trigger this if they block, such as calling C extension code that holds the GIL and takes a long time. A common misunderstanding comes from the different behavior, the fact that it's effectively also a request timeout in synchronous workers.

A graceful shutdown for synchronous workers, as written, should probably be set equal to the worker timeout. But if we were to have a request timeout, I'd probably like to reevaluate when workers close sockets and stop accepting requests, which currently happens immediately. In an environment like Kubernetes, that's not ideal because the infrastructure knows to drain the requests and it just needs time to react, time during which gunicorn should still accept new requests that arrive. Each new request would be subject to the request timeout, and a proper graceful timeout would be <time for infrastructure to react> + <request timeout>.

It's probably worth consulting what other popular projects are doing and gathering up some ideas somewhere.

[katzlbt]

Let me try to specdoc:

--request-timeout 30+10 or --request-timeout 30
Specification and Documentation: This option raises the SysExit(or any other like that) exception in the worker's user-process after 30s. This can be caught using except SysExit(e.g): to save intermediate results. Handling this exception by user-code may take +10 extra seconds until the worker or thread is killed without additional grace-period for a total of 40s max.

Backwards compatibility could be: --timeout 30 will be internally rewritten to --request-timeout 30 and documentation dropped.

My expected behavior is implemented in the arbiter.Arbiter.stop() method if graceful=True, I think (just code review), but not like it played out on sync.

gunicorn/gunicorn/arbiter.py

Line 391 in b5d78e8

limit = time.time() + self.cfg.graceful_timeout

PS: Happy holidays and new year! Be aware, that you can prevent further issues by updating the docs, but not by answering here. ;-)