Bug: session log corruption — resumed writer re-appends overlapping seqs at byte-EOF (appendLines never validates the durable tail)

[xdongHo]

Summary

Across three real sessions on Windows (dsh 0.1.0-rc.6, zstd-compressed JSONL store), history loading started failing with:

corrupt session log: seq gap in committed region at line 18839 (expected 131091, got 131087)
corrupt session log: seq gap in committed region at line 5 (expected 4, got 3)
corrupt session log: seq gap in committed region at line 45244 (expected 354321, got 354320)

All three are the same defect: when a session is interrupted/closed and then resumed, the interrupted-close block that a prior lifecycle persisted (synthetic closers from interruptedTurnClosers, plus the re-projected session/end-seed) is re-written from its starting seq by the resumed writer, while appendLines() appends at byte-EOF without validating the file's actual tail seq. The seq-continuity contract (event.seq === state.cursor + i in appendCore) is enforced only against the in-memory cursor, so any handoff where the cursor and the durable file diverge produces a durable seq overlap. The strict loader (SessionLogScanner.consumeEventLine) then rejects the entire stored history ("history unavailable" in the web UI).

Evidence (frame-level analysis of the real logs)

Large session (16k frames):

time	batch appended	seqs
T1	one frame: interrupted tool/result placeholder, step/end, turn/end(interrupted)	131087..131089
T1+24s	one frame: session/end-seed	131090
T1+24.2s	resumed writer: real tool result of the same call, step/end	131087..131088
...	continuation	131089..147264

The resumed batch passes the in-memory cursor check because its writer's cursor was 131087 — a state read before the close block was persisted — while the file already contained 131087..131090. appendLines then appends at EOF (after the marker) and the committed region overlaps.

Second session: stored session/end-seed at seq 3; the resumed writer's first batch starts at seq 3 (agent/inbox/spliced) and continues 3..40421. Same shape, one event of overlap.

Third session: identical signature — session/end-seed at seq 354320 superseded by the resumed agent/inbox/spliced at seq 354320, continuing to 378636. Four occurrences across three days on three sessions, always at an interrupt/resume boundary.

Root cause

• SessionPersistenceCoordinator.appendCore validates event.seq === state.cursor + i — in-memory only.
• JsonlSessionPersistence.appendLines opens the log with mode "a" and writes; there is no check that the durable tail actually continues at the batch's first seq.
• The close path persists synthetic closers + the end-seed marker at seqs that the resume path considers un-persisted (its in-memory log predates them). Any divergence between a lifecycle's cursor and the file (server restart, handoff, cached prepared source, plugin reload) lands here.

Suggested fix (validated locally)

Make the durable boundary seq-aware. In appendLines, before opening the append handle, decode the log's tail from its bytes (frame granularity for zstd, line granularity for plain) and classify the batch's first seq:

• firstSeq === lastSeq + 1 → clean append, unchanged behavior (fast path: only the last frame is decoded; measured 3.6 ms on a 4.1 MB / 16k-frame log);
• firstSeq <= lastSeq → the file tail is superseded by this writer's branch: truncate at the frame boundary ending at firstSeq - 1 (reusing the existing repair() truncate+fsync), then append. The writer's in-memory log is the authority for content; this converges the file and makes the corruption self-healing;
• firstSeq > lastSeq + 1, or the boundary falls inside a packed storage row → refuse loudly (the file stays valid).

We run this patch locally (against 0.1.0-rc.6): all three real corruption scenarios are healed automatically on the first post-resume append, clean-append behavior is byte-identical, torn tails are truncated as a side effect, and a 23-case integration suite passes (gap refusal, straddle refusal, idempotent rewrite convergence, zstd + plain layouts). Happy to share the patch or open a PR if contributions are accepted.

Until then, affected logs can be repaired offline by byte-level frame surgery: drop the superseded frames between the pre-overlap boundary and the resumed continuation, revalidate strict seq contiguity, and atomically replace the file.

Environment

• dsh 0.1.0-rc.6, profile web, Windows 11 (win32), Node 22.22.2
• store: session.jsonl.zstd per-session files under <home>/.dsh/sessions/<project-slug>/<session-id>/

[argszero]

Verified your mechanism against current upstream master (4e84901 = 0.1.2-alpha.4) — every claim holds, and this is the same root cause as the thread opened two days ago:

#5255 — "Session logs corrupted by concurrent writers: append path trusts its in-memory cursor" (2026-08-31). Your three-session Windows evidence plus Tazio7's macOS rc.2 recurrence on #2817 are the strongest cross-platform confirmations of that root cause yet, so it's worth consolidating here rather than re-deriving.

Source-level confirmation at alpha.4:

1. appendLines is a blind byte-EOF append. packages/session/session-persistence-jsonl/src/index.ts :693-717 does open(path, 'a') → writeFile(content) → sync(). It never reads the file's last durable line/frame back and never checks that events[0].seq continues the physical tail. The only failure handling is a rollback if the write itself throws (:708-716) — correct for partial writes, irrelevant to stale-tail overlap.

2. The seq-continuity contract is enforced only against the in-memory cursor. packages/session/session-persistence/src/coordinator.ts :783-790 validates event.seq === state.cursor + i, where state.cursor is set once at adoption/commitPrepared (:1116/:1135) and then advanced purely in memory per append (:799). Nothing re-derives it from the durable file between adoption and a later append.

3. The revision-convergence machinery protects only the cold prepare path. The prepare/load loops (:810-860) plus isPreparedSourceCurrent (:1144+) re-check the durable revision before committing a prepared source. But the live-append path (append → appendBatch → appendLines) never re-checks it. So once a session is adopted with a stale base — the retiring process's drain landed after your read — every subsequent append keeps validating against that stale cursor and lands at byte-EOF on top of the drain. Your 24 s and 3.4 h gaps show this is not a flush race; the writer is simply never told the file advanced.

4. Synthetic closers make the overlap deterministic, not racy. prepareCore (:1079-1081) always synthesizes missing interruptedTurnClosers, and commitPrepared (:1123-1127) durably appends them via commitRepair whenever the loaded log ends mid-turn. A resumer whose read predates the retirer's drain will therefore re-persist the same closer block from the same starting seq — matching your "interrupted-close block re-written from its starting seq" observation. Read-side already fails loud (that is the corrupt session log: seq gap error you see); the gap is entirely writer-side.

Fix direction (writer-side, one seam): at the live-append boundary, re-validate the durable tail before appending — cheapest is a first-line/tail-frame re-read (readers already do bounded tail recovery) or a durable-revision token re-check on every appendBatch; on mismatch, re-adopt from the physical tail instead of appending over it. The closers path additionally wants idempotence: only persist synthesized closers if the durable tail actually lacks them.

If you can share the raw tail around one overlap (zstd frame boundary + seqs), it would discriminate the two sub-cases (late drain after adopt vs. closer re-write) — useful for the regression fixture.