[bug] After a force-kill of the DSH host (deadlocked agents) + restart, user input appears late or never - write-behind batching loses the un-flushed tail

[qyz7438]

Bug report: after a manual force-kill of the DSH host (parallel agents deadlocked, process killed + restarted), user input appears late or never — session/message reload after restart does not recover what was in flight

Environment

• DeepSeek Harness Web GUI (dsh web, 127.0.0.1:3080)
• HEAD 47f943859b
• Related (earlier submissions): [bug] User text input delayed by a long time under heavy concurrent load (queued behind a busy agent turn - no pending feedback) #477 ("user text input delayed by a long time under heavy concurrent load") and [bug] Under heavy load new user requests no longer appear in the conversation/request view at all - history invisible (gap-repair strands user/message events) #479 ("under heavy load new user requests no longer appear in the conversation/request view at all").

Confirmed background (this report's root cause context)

• The DSH harness was manually force-killed while ~18 parallel agents were deadlocked / spinning (a hard kill of the process, not a graceful shutdown), then restarted.
• The reported symptoms — input appearing much later, and / or the user's own prior requests no longer visible in the GUI (history invisible) — occurred around and after that kill/restart.

Key finding: a force-kill drops every event still in the durable write-behind buffer

Session contents are persisted with a bounded write-behind batch in session-persistence (packages/session/session-persistence):

• coordinator.ts DEFAULT_WRITE_BATCH_MAX_DELAY_MS = 200 (line 30). Appended events are not written immediately; they sit in an in-memory pending queue until a per-session timer fires the durable write (write-behind.ts line 45 enqueue, line 82 armTimer, line 143 startWrite → options.write, which fsyncs).
• The only paths that drain pending synchronously to quiescence are flush() (write-behind.ts lines 63–72, drainBarrier) and the coordinator's explicit disposer drain (coordinator.ts lines 1091, 1139 observe disposed sessions).
• A hard kill/force-kill never runs flush() or the disposer drain. Any events still in pending — or in the middle of an in-flight fsync — at the moment the process died are lost from durable storage. They only existed in memory.
• Under the reported deadlock (18 spinning agents), the Node event loop was CPU-saturated, so the 200 ms deadline was routinely exceeded by far; a large volume of turns/messages across all 18 sessions could be sitting un-flushed when the kill landed.

The JSONL/SQLite backends then reload only the committed prefix on restart:

• JSONL (session-persistence-jsonl): readZstdPrefix (index.ts 348–419) decodes complete frames and recovers only the torn-tail committed events; the crash-repair (commitRepair, repair) truncates to the last complete frame and discards the un-committed in-flight tail. Events that never flushed are simply absent.
• SQLite (session-persistence-sqlite): test "an interrupted turn ... is PRESERVED and closed during load" / "a corrupt-JSON row in the uncommitted tail is discarded on load" (sqlite.spec.ts 258–350) confirms that after a crash, un-committed tail rows are dropped and the load rebalances to a committed prefix.

Why this explains the reported symptoms

1. "History invisible / can't see what I wrote before" (matches [bug] Under heavy load new user requests no longer appear in the conversation/request view at all - history invisible (gap-repair strands user/message events) #479 escalation): After restart the web GUI re-loads each session's messages from the committed prefix. Any user message or turn that was in the write-behind buffer (or a torn in-flight frame) at kill time is permanently gone from the durable log, so the GUI correctly shows nothing for it — but the user had typed/seen it, so it looks like their history vanished. This is durable data loss caused by the force-kill + write-behind batching, not just a live-stream rendering bug.

2. "Input appears much later" (matches [bug] User text input delayed by a long time under heavy concurrent load (queued behind a busy agent turn - no pending feedback) #477): After restart the harness is cold: sessions are re-materialized from disk, the browser reconnects (mux/host WebSockets close on host death and the client re-pulls session.list + history on reconnect — packages/client/runtime/src/client/sessions/manager.ts re-pull-on-reconnect), and the agent queue is rebuilt from the (truncated) durable log. Until that recovery completes, new input is queued behind the re-loading/busy loop, and the reconstituted turn numbering may re-order/steer queued messages, so a typed message can surface much later or be re-delivered differently than the user expected.

3. The restart is likely the transition that turned "delayed" into "never visible": the in-flight tail is lost at the kill, and the same write-behind + gap-repair mechanics analyzed in [bug] User text input delayed by a long time under heavy concurrent load (queued behind a busy agent turn - no pending feedback) #477/[bug] Under heavy load new user requests no longer appear in the conversation/request view at all - history invisible (gap-repair strands user/message events) #479 (session.ts acceptLiveEvent/repairGap, agent-loop/agent.ts serial turn) operate on a now-truncated, lower-seq log, so subsequent live events can carry the wrong expectations and strand or reorder after restart.

Relationship to the prior two reports

• [bug] User text input delayed by a long time under heavy concurrent load (queued behind a busy agent turn - no pending feedback) #477 (delay) and [bug] Under heavy load new user requests no longer appear in the conversation/request view at all - history invisible (gap-repair strands user/message events) #479 (no appearance) are best understood as secondary symptoms of the same recovery path after the force-kill+restart, under load.
• The durable data model is crash-safe for the committed prefix, but by design does not promise durability for the last ≤200 ms (and longer under CPU starvation) of appended events. A force-kill loses that tail. Nothing in the GUI or the reload path tells the user that the last messages before the kill were never persisted — so those messages just never come back.

Expected behavior

• Any message the user actually typed must never silently disappear after a restart. Even if a force-kill can lose up to ~200 ms (or more under load) of un-flushed events, the product should (a) shrink/minimize the loss window (e.g. flush user-input boundaries immediately rather than on a 200 ms timer), (b) mark a session's tail as "truncated by an unclean shutdown" when the committed prefix is short of what was running, and (c) surface that state in the GUI instead of presenting incomplete history as if complete.
• On reconnect/restart, the GUI should clearly indicate "reloading / recovering session" and show any locally-known-but-not-persisted input as a recoverable draft / pending item, rather than silently dropping it.
• The write-behind default feels too large under CPU starvation; consider a per-append immediate flush for user-origin user/message events (the most valuable content) and turn/step boundaries, and a bounded cap plus backpressure so a deadlocked overload cannot grow an unbounded in-memory loss window.

Suggested areas to look at

• packages/session/session-persistence/src/write-behind.ts and coordinator.ts — durability of the batching tail; flush-on-user-input; loss-window reduction and explicit "tail truncated" signal.
• packages/session/session-persistence-jsonl/src/index.ts (readZstdPrefix, commitRepair, repair) and session-persistence-sqlite — surfacing truncated-tail state to the web layer.
• packages/client/runtime/src/client/sessions/manager.ts (reconnect re-pull) and session.ts (acceptLiveEvent/repairGap) — recovery UX (reloading indicator, recoverable pending drafts) and gap-repair robustness after restart.
• Web GUI reload path: how a session's message window is rebuilt from session.history after reconnect (should mark missing tail from unclean shutdown).

Happy to attach the actual .zstd session logs (pre/post kill) if that helps confirm the exact truncated commit point.