Don't block server shutdown on idle keep-alive connections - #108581
Open
george-larionov wants to merge 1 commit into
Open
Don't block server shutdown on idle keep-alive connections#108581george-larionov wants to merge 1 commit into
george-larionov wants to merge 1 commit into
Conversation
On shutdown `server.stop` only closes the listening socket; a handler
thread already parked in `HTTPServerSession::hasMoreRequests` (a socket
read bounded by `keep_alive_timeout`) is not woken, so `server_pool.joinAll`
blocks for the full `keep_alive_timeout`. The interserver ("servers for
tables") phase has no force-close fallback, unlike the client phase's
`safeExit`, so an idle keep-alive connection there stalls shutdown for the
whole timeout.
Make the keep-alive wait interruptible: `hasMoreRequests` polls in 1s
slices and re-checks a stop callback (`!tcp_server.isOpen()`) between them,
so an idle connection observes shutdown within ~1s. Buffered and in-flight
requests are untouched.
Add an integration test that holds an idle keep-alive connection on the
interserver port and asserts shutdown still completes quickly.
Contributor
|
Workflow [PR], commit [f1c63c8] Summary: ❌
AI ReviewSummary
Missing context / blind spots
Final Verdict
|
clickhouse-gh
Bot
added
the
pr-bugfix
Contributor
LLVM Coverage Report
Changed lines: Changed C/C++ lines covered by tests: 4/4 (100.00%) | Lost baseline coverage: none · Uncovered code |
Member
Author
|
@vdimir would you like to review? |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
On shutdown,
server.stoponly closes the listening socket. A handler thread already parked inHTTPServerSession::hasMoreRequests— a socket read bounded bykeep_alive_timeout— is not woken, andHTTPServerConnection::runonly re-checkstcp_server.isOpen()after that read returns. The interserver ("servers for tables") shutdown phase then blocks inserver_pool.joinAllfor the fullkeep_alive_timeout, since (unlike the client-connection phase, which callssafeExit) it has no force-close fallback. Any idle keep-alive connection on such a server stalls shutdown for the whole timeout.Fix: make the keep-alive wait interruptible.
hasMoreRequestspolls in 1-second slices and re-checks a stop callback (!tcp_server.isOpen()) between slices, so an idle connection observes shutdown within ~1s. The wait is only shortened while idle — buffered and in-flight requests are handled before it — so no traffic is truncated.Added integration test
test_keep_alive_shutdownthat holds an idle keep-alive connection on the interserver port withkeep_alive_timeout=50and asserts shutdown still completes quickly. Verified: it fails (Server did not stop within timeout) without the fix and passes with it.Tradeoffs:
HTTPServerSession), extending its existing keep-alive machinery.pollthat returns immediately) — negligible per connection, a minor change in a hot path at very high idle-connection counts.Changelog category (leave one):
Changelog entry (a user-readable short description of the changes that goes into CHANGELOG.md):
Fixed server shutdown blocking for up to
keep_alive_timeoutseconds when an idle keep-alive HTTP connection (for example, an interserver connection) is open.Documentation entry for user-facing changes