Skip to content

Session wedges permanently (stuck "replying", no messages) when provider event stream stalls #46

Description

@saucam

Summary

A session becomes permanently stuck in "replying" with no messages displayed (both Telegram and web UI) when the provider event stream stalls without a terminal event — e.g. a Bash command or an MCP gateway call that never returns. The session is unrecoverable through normal sends; only an explicit interrupt (or destroy) frees it.

Not a regression — confirmed present on main as well as fix/file-explorer-session-switch; the multi-provider refactor inherited it unchanged. Severity is high regardless: any single hung tool/MCP call bricks the session.

Reproduction (observed live)

Session codeoid-more-bugs wedged at tool_running:

  • Last transcript record was a Bash tool_call with no following tool result.
  • SDK subprocess alive 8h+ but sleeping (Sl, ~0% CPU) — stalled, not working.
  • meta.lastActivityAt was later than the last transcript write — a status update fired with no message.
  • codeoid interrupt <name> returned it to idle and made it usable again.

Root cause

Two cooperating defects:

1. No stall watchdog on the turn event loop

Session.#consumeEvents (src/daemon/session.ts:1586) blocks in for await (const event of run.events) until a terminal turn_done/error. The only timeout anywhere in the loop is a 5s ZeroID fence (session.ts:1739); there is no watchdog on the overall stream. If the SDK subprocess stalls and never emits a terminal event, the loop blocks forever → the finally (session.ts:1630) never runs → #activeRun stays set and #status stays tool_running indefinitely.

2. Sends are silently swallowed into the dead run

With the session stuck "working", every subsequent send hits the mid-turn fast-path in #sendInner (src/daemon/session.ts:690):

if (wasWorking && this.#activeRun?.pushMidTurn) {
  ... this.#activeRun.pushMidTurn(...); this.#setStatus("thinking"); return;
}

It pushes the message into the wedged run and returns without starting a new turn. The user sees their own message + "⎆ Queued mid-turn", then nothing — forever. There is no liveness check on #activeRun before trusting it.

Aggravating factor: keep-warm interrupt leaves a hung subprocess alive

TurnRun.interrupt() (src/daemon/providers/claude/index.ts:152) calls the SDK's graceful q.interrupt(), which is keep-warm by design and does not kill the subprocess (only teardown() aborts it, line 189). So even after a manual interrupt, a hung subprocess is left warm and the next send can re-wedge on it.

Proposed fix

  1. Stall watchdog in #consumeEvents — drive the async iterator manually and race next() against a configurable stall timeout (generous default, e.g. 300s of complete silence; long-running tools still emit tool_progress/partial events, so true silence is a reliable hang signal). On stall: emit a clear system message, hard teardownProvider() (abort → reap subprocess, not warm interrupt), reset status to idle/error, break so finally clears #activeRun.
  2. Liveness guard in #sendInner — track #lastEventAt; before taking the wasWorking mid-turn-push branch, if the run has produced no event within the stall window, treat it as dead: tear it down and start a fresh turn instead of swallowing the message. (Auto-recovers on the user's next send, even before the watchdog fires.)
  3. Make the timeout configurable (config.session.turnStallTimeoutMs, default 300000).

Acceptance

  • A turn whose stream stalls (no events) auto-recovers within the timeout: status returns to idle, a clear message is shown, the subprocess is reaped.
  • After a stall, a new send starts a fresh turn and gets a reply (no permanent wedge).
  • Regression test (via MockSessionProvider): a run that never emits turn_done triggers watchdog recovery; a subsequent send completes normally.
  • Long-running legitimate tools (emitting tool_progress) are NOT killed.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingrobustnessProduction-grade robustness gaps

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions