Summary
A session becomes permanently stuck in "replying" with no messages displayed (both Telegram and web UI) when the provider event stream stalls without a terminal event — e.g. a Bash command or an MCP gateway call that never returns. The session is unrecoverable through normal sends; only an explicit interrupt (or destroy) frees it.
Not a regression — confirmed present on main as well as fix/file-explorer-session-switch; the multi-provider refactor inherited it unchanged. Severity is high regardless: any single hung tool/MCP call bricks the session.
Reproduction (observed live)
Session codeoid-more-bugs wedged at tool_running:
- Last transcript record was a
Bash tool_call with no following tool result.
- SDK subprocess alive 8h+ but sleeping (
Sl, ~0% CPU) — stalled, not working.
meta.lastActivityAt was later than the last transcript write — a status update fired with no message.
codeoid interrupt <name> returned it to idle and made it usable again.
Root cause
Two cooperating defects:
1. No stall watchdog on the turn event loop
Session.#consumeEvents (src/daemon/session.ts:1586) blocks in for await (const event of run.events) until a terminal turn_done/error. The only timeout anywhere in the loop is a 5s ZeroID fence (session.ts:1739); there is no watchdog on the overall stream. If the SDK subprocess stalls and never emits a terminal event, the loop blocks forever → the finally (session.ts:1630) never runs → #activeRun stays set and #status stays tool_running indefinitely.
2. Sends are silently swallowed into the dead run
With the session stuck "working", every subsequent send hits the mid-turn fast-path in #sendInner (src/daemon/session.ts:690):
if (wasWorking && this.#activeRun?.pushMidTurn) {
... this.#activeRun.pushMidTurn(...); this.#setStatus("thinking"); return;
}
It pushes the message into the wedged run and returns without starting a new turn. The user sees their own message + "⎆ Queued mid-turn", then nothing — forever. There is no liveness check on #activeRun before trusting it.
Aggravating factor: keep-warm interrupt leaves a hung subprocess alive
TurnRun.interrupt() (src/daemon/providers/claude/index.ts:152) calls the SDK's graceful q.interrupt(), which is keep-warm by design and does not kill the subprocess (only teardown() aborts it, line 189). So even after a manual interrupt, a hung subprocess is left warm and the next send can re-wedge on it.
Proposed fix
- Stall watchdog in
#consumeEvents — drive the async iterator manually and race next() against a configurable stall timeout (generous default, e.g. 300s of complete silence; long-running tools still emit tool_progress/partial events, so true silence is a reliable hang signal). On stall: emit a clear system message, hard teardownProvider() (abort → reap subprocess, not warm interrupt), reset status to idle/error, break so finally clears #activeRun.
- Liveness guard in
#sendInner — track #lastEventAt; before taking the wasWorking mid-turn-push branch, if the run has produced no event within the stall window, treat it as dead: tear it down and start a fresh turn instead of swallowing the message. (Auto-recovers on the user's next send, even before the watchdog fires.)
- Make the timeout configurable (
config.session.turnStallTimeoutMs, default 300000).
Acceptance
- A turn whose stream stalls (no events) auto-recovers within the timeout: status returns to
idle, a clear message is shown, the subprocess is reaped.
- After a stall, a new send starts a fresh turn and gets a reply (no permanent wedge).
- Regression test (via
MockSessionProvider): a run that never emits turn_done triggers watchdog recovery; a subsequent send completes normally.
- Long-running legitimate tools (emitting
tool_progress) are NOT killed.
Summary
A session becomes permanently stuck in "replying" with no messages displayed (both Telegram and web UI) when the provider event stream stalls without a terminal event — e.g. a
Bashcommand or an MCP gateway call that never returns. The session is unrecoverable through normal sends; only an explicitinterrupt(ordestroy) frees it.Not a regression — confirmed present on
mainas well asfix/file-explorer-session-switch; the multi-provider refactor inherited it unchanged. Severity is high regardless: any single hung tool/MCP call bricks the session.Reproduction (observed live)
Session
codeoid-more-bugswedged attool_running:Bashtool_callwith no following tool result.Sl, ~0% CPU) — stalled, not working.meta.lastActivityAtwas later than the last transcript write — a status update fired with no message.codeoid interrupt <name>returned it toidleand made it usable again.Root cause
Two cooperating defects:
1. No stall watchdog on the turn event loop
Session.#consumeEvents(src/daemon/session.ts:1586) blocks infor await (const event of run.events)until a terminalturn_done/error. The only timeout anywhere in the loop is a 5s ZeroID fence (session.ts:1739); there is no watchdog on the overall stream. If the SDK subprocess stalls and never emits a terminal event, the loop blocks forever → thefinally(session.ts:1630) never runs →#activeRunstays set and#statusstaystool_runningindefinitely.2. Sends are silently swallowed into the dead run
With the session stuck "working", every subsequent send hits the mid-turn fast-path in
#sendInner(src/daemon/session.ts:690):It pushes the message into the wedged run and returns without starting a new turn. The user sees their own message + "⎆ Queued mid-turn", then nothing — forever. There is no liveness check on
#activeRunbefore trusting it.Aggravating factor: keep-warm interrupt leaves a hung subprocess alive
TurnRun.interrupt()(src/daemon/providers/claude/index.ts:152) calls the SDK's gracefulq.interrupt(), which is keep-warm by design and does not kill the subprocess (onlyteardown()aborts it, line 189). So even after a manual interrupt, a hung subprocess is left warm and the next send can re-wedge on it.Proposed fix
#consumeEvents— drive the async iterator manually and racenext()against a configurable stall timeout (generous default, e.g. 300s of complete silence; long-running tools still emittool_progress/partial events, so true silence is a reliable hang signal). On stall: emit a clear system message, hardteardownProvider()(abort → reap subprocess, not warm interrupt), reset status toidle/error, break sofinallyclears#activeRun.#sendInner— track#lastEventAt; before taking thewasWorkingmid-turn-push branch, if the run has produced no event within the stall window, treat it as dead: tear it down and start a fresh turn instead of swallowing the message. (Auto-recovers on the user's next send, even before the watchdog fires.)config.session.turnStallTimeoutMs, default 300000).Acceptance
idle, a clear message is shown, the subprocess is reaped.MockSessionProvider): a run that never emitsturn_donetriggers watchdog recovery; a subsequent send completes normally.tool_progress) are NOT killed.