Stop a restart from inventing work a session isn't doing - #1171
Merged
Conversation
A daemon restart that reattaches to an adapter which outlived it was rewriting the session's state to Running. Nothing about that adapter changed across the restart: a session parked at its harness's prompt is still parked there, blocked on input, with no reason to emit another status. For a PTY harness the idle sweep eventually walks it back, because silence is measurable against output. A headless session emits none between turns, so the false Running was unfalsifiable — it survived until a client happened to open the session and the hydration resize forced the child to repaint. Until then the fleet painted a working spinner over an idle session, and the busy span banked hours of compute the session never spent. So an idle session now comes back idle. Every other state still resumes as Running, which is what a live adapter means for them, and the spawn path — where a replacement child really is booting — is untouched.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A headless Slack service session shows an animating running icon after every daemon restart, with no turn in flight. Clicking into it stops the animation and it drops to idle.
What's happening
respawnsetsSessionState::Runningon both of its branches — including the one where the adapter outlived the daemon and the daemon merely reattached to it. Nothing about that adapter changed across the restart. It is blocked in the sameinbox.recv()it was blocked in before, and an adapter waiting for input has no reason to announce anything: its next status arrives when a turn ends, which is to say never, until someone starts one.That makes the optimistic
Runningunfalsifiable:last_pty_at_msto measure silence against.Noneand the sweep never fires.state == Runningalone.pty_resize(with the +1 column bump), the child repaints, PTY bytes finally arrive — and now quiescence flips it toAwaitingInput. Hence "clicking it fixes it."Meanwhile
set_state_trackedhad opened a busy span on that falseRunning, so the session banked its idle hours as compute. The session in the report was sitting on ~6.7M ms of it.The fix
A reattached session keeps the state it was persisted with —
AwaitingInputstaysAwaitingInput. Every other state still resumes asRunning, which is what a live adapter means for them:Runningwas mid-turn and still is,Pendingnever reported anything, andErroredmust stop looking terminal now that its adapter answered.The spawn path is deliberately untouched. There a replacement child really is booting, so the placeholder is honest — and marking it
AwaitingInputearly would re-open the spec-0149 hazard of input pasted into a still-drawing harness.Tests
reattach_leaves_an_idle_session_idle— unit-covers the state mapping.restart_keeps_an_idle_session_idle— e2e: creates a session, waits for it to park, restarts the daemon, then samples state every 100ms for 3s and requires idle at every sample. Verified it fails on the old behavior (left: Running, right: AwaitingInput) and passes with the fix.cargo testgreen. (service_supervisor::…hand_a_port_to_another…flaked once under concurrent load, passes 3/3 on rerun; it's a port-binding test, unrelated to lifecycle.)specs/0180-a-restart-does-not-invent-work.mdrecords the rule: resume must treat "the adapter is alive" and "the session is working" as different facts.