[Triage] gthread evolution: lifecycle of an accepted connection

[aszubarev]

Disclaimer

This discussion focuses on a suggestion to improve the lifecycle of an accepted connection. But to understand it you first need to consider the evolution of the gthread worker. An improvement suggestion will be described at the end

The evolution of gthread

v20.1.0: gthread deadlock

The diagram below shows the architecture of the gthread worker version v20.1.0. It can be divided into three parts: the event loop, request processing, and graceful shutdown.

At the beginning, the listen-socket is switched to non-blocking mode. Then the callback accept is registered with the poller. After that, an infinite loop is started, which waits for events from the poller with a timeout of 1 sec. If no events are received during this interval, the loop performs any final operations and then waits again.

When a connection request is received, the callback accept is triggered. Inside it, the client socket is made blocking. The request processing task is submitted in the thread pool, and the callback finish_request is registered on the Future instance.

When a pool thread starts handling the request, it reads data from the client in blocking mode. After receiving all the data from the client, the request counter (nr_conns) is compared with the --max-requests value. If the counter exceeds this threshold, the worker’s alive status is set to False. This causes the main thread to exit the event loop, after which the graceful shutdown procedure is started. It is important to note that the processing of the request itself continues in a dedicated pool thread.

When processing is completed, the finish_request callback is called in the same thread. It determines whether to keep the connection or close it immediately. If the connection is kept, the client socket is switched to non-blocking mode, a timeout is set, and another callback (reuse_connection) is registered with the poller. The only difference between reuse_connection and accept is that reuse_connection does not establish a new connection; otherwise, their behavior is identical.

The main conclusion is that reading data from the client blocks the thread. As soon as the server accepts the connection, the task is immediately added to the pool. If the client establishes a connection but does not transmit data, the thread will be blocked waiting for packets. This is what caused the issue «Gunicorn gthread deadlock #2917». This behavior was expected, and the official deployment guide even recommends running the application server behind a reverse proxy. Later, the pull request «gthread: only read sockets when they are readable #2918» was accepted and included in the v21.0.0 release.

v21.0.0: resilience to speculative connections

The main goal of the changes is protection against speculative connections. It was decided to submit the request processing task into the thread pool only after the client actually starts sending data. The architecture of version v21.0.0 is as follows:

Yes, this solved the problem of speculative connections, but along with the new architecture came new bugs. Users widely encountered the issue “Connection reset during max-requests auto-restart with gthread #3038”: the client receive 502 errors.

This issue occurs in the following cases:

• Reload the configuration
• Upgrading to a new binary on the fly
• Automatic restart after --max-requests
• Stopping the old instance when deploying a new version.

The cause of this issue is a specificity of the graceful shutdown.

class ThreadWorker(base.Worker):

    def run(self):
        
        ...

        self.tpool.shutdown(False)
        self.poller.close()

        for s in self.sockets:
            s.close()

        futures.wait(
            self.futures, 
            timeout=self.cfg.graceful_timeout,
        )

After exiting the loop, the following steps are executed:

• Terminate the thread pool (without waiting)
• Close the poller
• Close the listen sockets
• Wait for request-processing tasks that have already been added to the pool

Closing the poller also closes all sockets registered in it. After a successful accept, the processing task was not submitted into the pool. Instead, the client socket was registered with the poller to wait for data. If at that moment the worker changed its alive status to False, then the just-accepted client socket was immediately closed. A client sending a request or reading a response would then receive a «Connection reset by peer» error.

A similar situation occurs with Keep-Alive connections. The server sets a timeout on them and registers them in the poller to wait for a new request. A client sending a request over such a connection assumes it is still alive, but the server may have closed it earlier, even if the timeouts are correctly configured.

v24.0.0: refactoring «Graceful Shutdown»

This release includes the following changes:

• Return to the previous accept logic: the request processing task is again submitted into the pool immediately after the connection is established
• Complete refactoring of «Graceful Shutdown»
• Introduction of PollableMethodQueue (see #3440)

The return to the previous accept logic fixes issue #3038. The new implementation of graceful shutdown ensures that keep-alive connections are properly terminated.

class ThreadWorker(base.Worker):

    def run(self):
        
        ...

        # Graceful shutdown: stop accepting but handle existing connections
        self.set_accept_enabled(False)

        # Wait for in-flight connections within grace period
        graceful_timeout = time.monotonic() + self.cfg.graceful_timeout
        while self.nr_conns > 0:
            time_remaining = max(graceful_timeout - time.monotonic(), 0)
            if time_remaining == 0:
                break
            self.wait_for_and_dispatch_events(timeout=time_remaining)
            self.murder_keepalived()

        # Cleanup
        self.tpool.shutdown(wait=False)
        self.poller.close()
        self.method_queue.close()

        for s in self.sockets:
            s.close()

After exiting the main loop, the listen sockets are unregistered from the poller. This ensures that no new connections will be accepted in the next «shutdown» cycle. However, as long as the server still has active connections, it waits for new requests to arrive on them.

In other words, gthread now follows the rule: if a timeout is set on a connection, the worker waits for a new request until that timeout expires. When the client and server timeouts are aligned, the close race disappears.

A side effect of this implementation is that the --keep-alive parameter directly affects the server's shutdown time: the longer this timeout, the longer graceful shutdown can take.

v25.2.0: prevent thread pool exhaustion

Release v24.0.0 was again affected by issue «Gunicorn gthread deadlock #2917». As a result, an interesting pull request «fix(gthread): prevent thread pool exhaustion from slow clients #3519» was accepted and included in release v25.2.0.

Worker threads now wait up to 5 seconds for data. If no data arrives, the connection is deferred back to the main poller, freeing the thread for other work.

Improvement suggestion

Instead of waiting for client data in a pool thread, we can wait for it in the main thread — an approach similar to what was used in versions v21.0.0 to v23.0.0. But this time, set a timeout for the connection. When the connection expires, it will be closed in the murder_pending method.

class ThreadWorker(base.Worker):

    def accept(self, listener):
        """Accept a new connection from a listener socket."""
        try:
            client_sock, client_addr = listener.accept()
            self.nr_conns += 1
            client_sock.setblocking(True)

            conn = TConn(self.cfg, client_sock, client_addr, listener.getsockname())

            conn_timeout = max(
                DEFAULT_PENDING_REQUEST_MIN_WAIT_TIMEOUT,  # 5 sec.
                self.cfg.keepalive,
            )
            conn.timeout = time.monotonic() + conn_timeout

            self.pending_conns.append(conn)
            self.poller.register(
                conn.sock,
                selectors.EVENT_READ,
                partial(self.on_pending_socket_readable, conn),
            )

Advantages:

• Also works with fast connection
• Also works during a graceful shutdown (thx v24+)
• Avoids creating temporary poller for prevent thread pool exhaustion
• No blocking of the thread even for 5 seconds!

The graceful shutdown refactoring in v24+ prevents issue #3038 from occurring. This makes it possible to submit the request processing task only when data is available.

Full implementation example #3635