Fix (partial): a session stuck reporting "Active session is closing" forever when its kernel is OS-stopped (#1072) #1795
kaluli123123
started this conversation in
Bug reports
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Scope note
This addresses one of #1072's three reported symptoms — "Active session
<id>is closing" persisting forever after reattach fails, with the underlying worker process observed stuck in OS stateT(job-control-stopped, e.g. an external SIGSTOP). The other two reported symptoms (idle-eviction-sweep timeouts leaking worker processes, and a session showing "Executing" after its tool call already finished) share the same general shape — something the daemon awaits assumes will always eventually settle, and a genuinely frozen (not merely slow) peer process breaks that assumption — but are separate leak points in different code paths and are not covered by this change. This is one focused sub-fix, not a full fix for #1072, and I want to be upfront about that rather than overclaim.Root cause
closeSession()(daemon-mode.ts) setsclosingSessions.set(activeSessionId, close)before awaitingclosePromise(→closeSessionOnce, which awaitsstate.runtime.session.abort()andstate.runtime.dispose()), and only clears that bookkeeping inclosePromise's ownfinallyblock. If the session's kernel process becomes OS-level job-control-stopped, it can no longer respond to anything short ofSIGCONT/SIGKILL, so whatevercloseSessionOnceis awaiting on it never settles, thefinallyblock never runs, andclosingSessionsnever clears — every future attach or close for that session fails with "is closing" until the whole worker/daemon is killed and the OS-level state is gone.Why not just add a timeout
This project deliberately does not use blanket timeouts for RPCs/waits that can legitimately run a long time — see PR #1701, "remove blanket RPC timeouts, reject on real boundaries." A naive fixed-duration timeout here would just as wrongly treat a slow-but-healthy close as failed. The fix instead checks the kernel process's actual OS state directly, so it only ever acts on a confirmed-frozen peer, never a merely-busy one.
Fix
utils/child-process.ts: newisStoppedProcess(pid), reusing the existing/proc-stat-then-ps-fallback pattern already used byisZombieProcess, checking for POSIX stateTspecifically.core/kernel/index.ts/core/tools/ipython.ts/core/agent-session.ts: a newkernelPid/kernelProcessPidgetter chain (KernelManager→IpythonKernelProvisioner, already held privately byAgentSession→ a publicAgentSession.kernelProcessPid) exposes the running kernel's OS pid for this kind of liveness check, without adding any new public surface to the Jupyter-protocol-facing parts of the kernel.modes/daemon/daemon-mode.ts:closeSession()now starts awatchForStoppedKernelDuringClose()watchdog alongside the close. It pollsisStoppedProcess()on the session's kernel pid every 2s while the close is still pending; on the first confirmed-stopped poll it sends oneSIGCONT(a session that was merely, legitimately suspended resumes and the close proceeds normally); if it is still confirmed stopped after further polls despite that, it releases theclosingSessionsentry (matching the existing entry by identity, the same checkcloseSession's own cleanup already uses, so a race between the two is harmless) so the session becomes reattachable again. The original close keeps running in the background and settles normally whenever/if the kernel resumes on its own.Tests
test/child-process.test.ts:isStoppedProcessagainst a real SIGSTOP/SIGCONT cycle on a real spawned subprocess (not mocked), confirming it is true only while genuinely stopped and distinguishes "stopped" from "dead"/"zombied"/"merely running".test/suite/regressions/1072-stopped-kernel-close-recovery.test.ts(new): drives the realcloseSession()path (not the watchdog method in isolation) with a deliberately-hungruntime.dispose()and a mockedisStoppedProcess/process.kill, using fake timers for determinism: confirmsSIGCONTis sent once,closingSessionsis released only after the configured number of confirmations, and (second test) a merely-slow (never reported stopped) close is left completely alone.Both verified to fail against pre-fix
daemon-mode.tsand pass after the fix.Validation
npx tsgo -p tsconfig.json --noEmitand the rootnpm run check(biome, tsgo, installer render, browser smoke) both pass. Added apackages/coding-agent/.changes/fragment per the changelog-fragment CI check.Patch
Branch: https://github.com/kaluli123123/prime-agent/tree/fix/stopped-kernel-close-recovery
Diff: main...kaluli123123:prime-agent:fix/stopped-kernel-close-recovery
All reactions