[Bug] Persistence pipeline wedges permanently after one failed drain batch — session becomes a silent zombie until service restart

[Phantomthedog]

[Bug] Persistence pipeline wedges permanently after one failed drain batch — session becomes a silent zombie until service restart

Summary

A single failed drainBuffered() batch in @deepseek-ai/dsh-session-persistence-jsonl sets drainPaused = true and the drain timer is never re-armed. From that moment the session's durable log never receives another byte while the process keeps running: all subsequent events (including step/end, turn/end and any later turns) accumulate in the in-memory buffer only. There is no retry, no backoff, and no warning surfaced to the user. The session keeps accepting messages, so the failure is completely silent until the user notices history is not being saved.

Environment

• DSH 0.2.1-rc.2, Windows native (node v24.14.0, win32)
• Deployment: DSH web service (dsh-native bootstrap, port 3080)
• Observed 2026-10-19

Forensic timeline of the observed incident (all times MDT)

1. 02:07:34 — turn 2 / step 65 starts; an LLM request goes out at 02:07:34.413.
2. 02:07:48 — the in-flight LLM request is aborted by the DSH side (upstream gateway logged status 499 "the agent canceled the request", in/out=0; TTFT 2.98s had already arrived). Cancel cause was one of user/parent/disposed (hook path excluded — provider is a custom magpie route, not deepseek-account).
3. 02:07:37 — last write to session.v4.jsonl.zstd (seq 1896 = step/start) and to the projection cache. From this point the session writes nothing to disk for the lifetime of the process (verified: zero file growth for ~60 minutes, while other sessions in the same process kept persisting normally).
4. 02:37:24 — turn/end is appended in memory only: the live event stream reaches seq 1912 vs. seq 1896 on disk; 16 events exist solely in the buffer.
5. Sending new messages afterwards produced "empty turns" (turn/start → step/start → turn/end, 4 events, zero LLM requests, zero disk growth) — the session is effectively a zombie until the DSH service is restarted.

Root cause (from the shipped lib)

@deepseek-ai/dsh-session-persistence-jsonl/lib/index.js:

• enqueueLive() (≈L187-194) pushes the event into this.buffered and arms a 200 ms timer that calls drainLive().catch(reportBackgroundFailure). append() never propagates errors to the caller (fire-and-forget by design).
• drainBuffered() (≈L205-221): on any persist failure it does this.buffered = batch.concat(this.buffered); this.drainPaused = true; throw error. drainPaused is only cleared at the top of a successful drain pass — but once set, enqueueLive() returns early without re-arming the timer, so no further drain is ever attempted unless some other path calls flush().
• persistContiguous() (≈L223-241): ensureLease() → assertContiguous → torn-tail repair → storage.persistBatch; any throw here triggers the wedge.

Net effect: one transient failure (antivirus lock, transient FS error, lease hiccup) permanently and silently kills durability for that session, while the agent loop continues normally in memory. reportBackgroundFailure does not surface anything user-visible in the default deployment.

Expected behavior

• A failed drain batch should be retried with backoff instead of parking forever; and/or
• drainPaused should re-arm on the next enqueueLive() after a cooldown; and/or
• the failure should be surfaced (agent/error event, UI badge, log line) so the user knows the session is no longer persisting.

Suggested fix directions

1. Re-arm the drain timer after a bounded backoff when drainPaused is set, keeping the failed batch at the head of buffered.
2. Emit a persistent "persistence degraded" warning (projection-cache field or event) the Web UI can display.
3. On session close/reload, attempt a final drain and log clearly if buffered events would be dropped.

Happy to provide the full forensic timeline and the decoded session log if useful.

[PerryLink]

The pause is real and the timer really is not re-armed. One correction before the write-up: there is a retry entry point, and the report is stronger if it says what is missing rather than saying nothing retries.

packages/session/session-persistence-jsonl/src/storage.ts:

276:    if (this.batchTimer !== undefined || this.drainPaused) return
277:    this.batchTimer = setTimeout(() => {
311:          this.drainPaused = true

So after a failed batch, enqueueLive returns before arming anything: new events pile into buffered and nothing will call the drain on its own. That part is exactly as you describe.

But :299 clears the flag at the top of the next drain attempt, and :288-291 exposes drainLive() for exactly that: the doc comment at :285-286 says "a failure retains the batch in order so session/flush can retry and reject loudly". flushAll() at :508-513 walks every writer and calls writer.drainLive(), and the service exposes it as sessionPersistence.flush() (index.ts:472).

So the accurate defect is: recovery depends on an external flush(). If anything in the runtime flushes that session after the failure, the buffer drains and the pause clears; if nothing does, the session stalls silently forever. Your timeline is consistent with the second case, and that is the sentence I would put at the top — "the only recovery path is a flush that nothing schedules" — because it tells a maintainer where to look instead of inviting the reply that flush() exists.

Two supporting details from the same code:

• reportBackgroundFailure is only wired to the deadline-driven timer (:279). A failure that happens on that timer is reported; the state it leaves behind is the pause, and nothing reports that state.
• Because :276 short-circuits on drainPaused before checking buffered, the buffer can grow without bound while paused. On a long session that is a second failure mode on top of the durability one.

The fix I would propose matches your framing: on failure, re-arm a bounded backoff timer instead of relying on the next flush, and surface the paused state (a warning, or a step/end-adjacent marker) so a session that stops persisting is visible before the user notices missing history.