fix: count an accepted connection in the drain before its task starts - #691
Merged
Conversation
a graceful drain promised that every accepted connection had finished. it did not keep that promise: `shutdown.enter()` ran inside the spawned connection task, so between the `spawn` and the task's first instruction a connection was invisible. a shutdown landing in that window saw nothing outstanding, `drain_default()` returned 0, and a client accepted moments earlier had its socket cut with no response and no trace. reproduced on both backends: accept one connection, spawn its task, drain immediately, and the drain reports a clean shutdown over a connection that was never served. the count now starts on the accept loop, in the same breath as the accept, and the task's existing `defer shutdown.leave()` is the other half of it. each of the six accept loops hands its socket over through a small named function that takes the concurrency permit and the drain count and then spawns — the permit was already taken there, so the two acquisitions now sit together, and the hand-off is a thing a test can drive. that leaves one direction to be wrong in, and it is the safe one. work that is counted and never runs holds its count up until the grace period expires, and `drain(deadline_ms)` already gives up at its deadline and returns what it abandoned. bounded and reported, against a connection dropped silently.
a tls accept loop released its listener's binding to the server config the moment it stopped accepting, before draining. the binding is how tls.handshake finds the certificate and the private key, and a connection that was accepted but has not reached its handshake yet still needs it — so that connection's handshake failed and the client got nothing, on the one path a graceful shutdown exists to make clean. the window was always there; counting a connection from the accept opened it wide enough to be hit reliably, and CI hit it. proved by delaying the connection task 200ms before its handshake, which is what a task that has been spawned and not yet scheduled looks like: without this change the handshake fails with "tcp_read_bytes failed", with it the connection is served through the drain. the two halves of the teardown are now split by when they are safe. the listening socket is closed before the drain, because a rolling deploy's replacement is waiting to bind that port and the drain can take the whole grace period. the config binding is dropped after the drain, next to the config close, because every connection that borrowed it has finished by then.
This was referenced Aug 10, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
a graceful drain promised that every accepted connection had finished, and did
not keep it.
shutdown.enter()ran inside the spawned connection task, sobetween the
spawnand that task's first instruction a connection wasinvisible: it had not entered,
drain_default()saw nothing outstanding andreturned, and a client accepted moments before the shutdown request had its
socket cut with no response and no trace.
the count now starts on the accept loop, in the same breath as the accept, and
the task's existing
defer shutdown.leave()is the other half of it. each ofthe six accept loops hands its socket over through a small named function that
takes the concurrency permit and the drain count and then spawns. the permit was
already taken on the loop, so the two acquisitions now sit together, and the
hand-off becomes a thing a test can drive — which is the only way to look at the
count in the window between a spawn and a task's first instruction.
that leaves one direction to be wrong in, and it is the safe one. work that is
counted and never runs holds its count up until the grace period expires, and
drain(deadline_ms)already gives up at its deadline and returns what itabandoned — bounded and reported, against a connection dropped silently.
the six loops were the whole surface. nothing else in std pairs
enter()withleave(), and the one other subsystem that registers with the drain,std.obs's exporter, already callsexpect_flush()before itsspawnratherthan inside the task.
what was tested
the bug first, on both backends. one connection accepted and spawned, then
drain_default()immediately:identical under
PITH_GREEN=0andPITH_GREEN=1— a clean drain reported overa connection that was never served. with eight connections the count read 2 of 8
on os threads and 1 of 8 under green, so most of a burst is invisible.
five new colocated tests, three in
std/net/http2/server.pith(h2c, h2cstreaming, tls) and two in
std/web.pith(plaintext, tls). each opens realsocket pairs, hands them to the accept loop's spawn function by hand, and reads
the count in exactly the window the bug lived in. the peers send nothing and
stay open, so no task can reach its
leaveand the expected count is exactrather than a race:
assert_eq(shutdown.inflight(), 8)right after the hand-offs — the count iscomplete when the last hand-off returns, however many tasks have started.
assert_eq(shutdown.drain(20), 8)— with nothing able to finish, the draingives up at its deadline and reports the work rather than hanging or claiming
a clean shutdown.
assert_eq(shutdown.drain(5000), 0)plusassert_eq(shutdown.inflight(), 0)— every task leaves exactly once, on thepath where the connection dies before its first frame or mid-handshake. no
double count, nothing left behind.
no sleeps anywhere: the listener is bound before anything connects to it, and
every step is ordered off
shutdown.inflight().each test was falsified. with
enter()moved back inside the slot functions allfive fail, on both backends:
and with the fix restored,
28 passed, 0 failedand25 passed, 0 failedonboth backends, plus
std/shutdown.pithat 17/17 on both.make run-regressions-only: 332 passed, 0 failed.docs:
docs/signals.mdgains a section on when a connection joins the count andwhy the accept is the right place,
std.shutdown'senter()doc says to call itbefore handing work to a task, and the module header describes the two halves
where they now live.
a second bug the first one uncovered
CI caught
test_tls_server_config_releasefailing under os threads, and it wasa real defect rather than a flake. a tls accept loop released its listener's
binding to the server config the moment it stopped accepting, before draining.
that binding is how
tls.handshakefinds the certificate and the private key,so a connection that had been accepted but had not reached its handshake yet
lost them: the handshake failed and the client got nothing, on the one path a
graceful shutdown exists to make clean.
the window was always there. counting a connection from the accept opened it
wide enough to be hit reliably — the same accept-time count made an existing
test into a real guard for it. the two halves of the teardown are now split by
when each is safe: the listening socket is closed before the drain, because a
rolling deploy's replacement is waiting to bind that port, and the config
binding is dropped after the drain, next to the config close, once every
connection that borrowed it has finished.
docs/tls.mdshowed the old orderingin its example and now shows this one.
proved by delaying the connection task 200 ms before its handshake — which is
what a task that has been spawned and not yet scheduled looks like:
make run-regressions-onlyre-run on top: 332 passed, 0 failed. the tls, http/2,websocket, grpc-shutdown and web cases were also run individually under both
backends.