Bug: session log corrupted on resume - last 4 committed events re-appended with fresh timestamps (seq gap in committed region)

[JudgeAllen]

Summary

Resuming a persisted session can append a stale 4-event prefix to its JSONL session log: the last four committed events are re-appended verbatim (same seq, same payload, fresh time), producing a seq gap in the committed region. The reader then hard-fails with corrupt session log: seq gap in committed region, the session history becomes permanently unloadable in the Web GUI, and the log requires manual surgery to repair.

Affected on 2026-08-15, dsh 0.1.0-rc.5, Windows, JSONL+zstd persistence backend (session.jsonl.zstd), 4 sessions with an identical signature.

Environment

• dsh version: 0.1.0-rc.5 (running profile packages and checkout HEAD 47f9438 all 0.1.0-rc.5)
• OS: Windows
• Persistence: session-persistence-jsonl with zstd compression (session.jsonl.zstd)
• Frontend: Web GUI (history load reports the error)

Symptom

Opening the session in the Web GUI:

history unavailable for session "session-9173e707-2020-4fba-96cd-55f056b74a9f": Error:
corrupt session log: seq gap in committed region at line 4988 (expected 93137, got 93133)(internal)

Evidence (4 real logs, all same signature)

session id	cwd	gap line	expected	got	re-appended (duplicated) seqs	duplicate event types
session-9173e707-2020-4fba-96cd-55f056b74a9f	F:\Git\dsh-plugins	4988	93137	93133	93133-93136	tool/result, step/end, step/start, assistant/chunk
session-2503d35c-ad47-4811-9ae6-f944c1dad356	F:\CommonTasks	1267	17961	17957	17957-17960	approval/decided, tool/result, step/end, step/start
session-6f6ddc38-9705-4cfd-b132-506c5881b5df	F:\Git\dsh-plugins	1110	14902	14898	14898-14901	tool/result, step/end, step/start, assistant/chunk
session-88728413-fa95-4993-8f10-5158eb91a881	F:\Git\dsh-plugins	612	5792	5788	5788-5791	tool/result, step/end, step/start, assistant/chunk

Every gap is exactly 4 events behind (got == expected - 4), and the duplicated events are exactly the last 4 events of the committed prefix, with identical types/payloads but fresh timestamps (~2 minutes after the originals). The file resumes at the correct seq immediately after the duplicate prefix and is otherwise perfectly contiguous to EOF (e.g. 9173e707 continues 93137?132739 with zero further discontinuities, ending in a completed turn/end).

Detailed trace for session-9173e707 (lines 4984-4992, 1 event per line)

L4984 seq=93133 tool/result      turn=2 step=46 t=1786795635084   ? committed (original run)
L4985 seq=93134 step/end         turn=2 step=46 t=1786795635084
L4986 seq=93135 turn/end         turn=2        t=1786795635084   (reason: interrupted)
L4987 seq=93136 session/end-seed               t=1786795746084   (+110 s: prepare/reseed)
L4988 seq=93133 tool/result      turn=2 step=46 t=1786795755580   ? RE-EMITTED (+9.5 s later)
L4989 seq=93134 step/end         turn=2 step=46 t=1786795755581
L4990 seq=93135 step/start       turn=2 step=47 t=1786795755618
L4991 seq=93136 assistant/chunk  turn=2 step=47 t=1786795756858
L4992 seq=93137 assistant/chunk  turn=2 step=47 t=1786795756859   ? correct continuation

The re-emitted sequence is exactly what an agent loop resuming an interrupted step emits first: re-deliver the in-flight tool/result, close the step, open the next step, stream the first assistant/chunk - but its seq counter was 4 behind the durable log (as if the session view was missing the last 4 committed events).

Root cause analysis

1. The append path never validates durable seq continuity. appendLines() in session-persistence-jsonl/src/index.ts encodes the batch and appends+fsyncs blindly; nothing checks that the batch's first event seq continues the file's committed prefix. Session.append only validates against the in-memory log, so a stale in-memory/snapshot view writes duplicate seqs without any error.
2. The resume path re-appends already-committed events. The 4-event tail re-emission with fresh timestamps (tool result ? step end ? step start ? first chunk) is a live-loop resume artifact, not a torn-tail retry (the torn-tail path only re-writes recovered bytes, and commitRepair writes synthetic closers - not this shape).
3. The reader cannot self-heal this case. SessionLogScanner treats a seq gap inside the committed region as damage, and since a turn/end exists after the gap it throws (format.ts line ~368). Torn-tail truncation does not apply because the gap is not at EOF. Result: the session is permanently unloadable until the log is manually rewritten.

Suggested fixes

1. Write-side invariant (primary): appendLines (or the coordinator append path) must verify the batch's first event.seq equals the durable committed event count (obtained via the scanner/committed prefix) and refuse - or truncate-and-repair - instead of writing a duplicate prefix. This turns the corruption into a loud error at write time, or prevents it entirely.
2. Resume path: ensure the resumed agent's seq counter derives from a fresh load of the committed prefix, never from a stale snapshot; any re-delivery of already-committed events (tool result replay for GUI/state reconstruction) must be surface-only and must not be appended durably.
3. Optional reader self-heal: when the gap is exactly a re-emission of the tail prefix (events identical except time), the scanner could drop the duplicates and keep the file readable - but prevention (1)/(2) is the real fix.

Manual repair performed (for reference)

Repair script dropped exactly the 4 duplicate lines (frames containing them were re-encoded; all other zstd frames kept byte-identical) and re-verified full contiguity with the harness scanner semantics (seq 0..N contiguous, ends turn/end, no torn frames). All 4 logs repaired; original corrupt bytes kept as session.jsonl.zstd.corrupt-bak beside each log. No real conversation content was lost (duplicates only).

[argszero]

Outstanding report — the 4-log evidence, the exact 4-behind signature, and the manual-repair detail make this one of the cleanest corruption-family write-ups we've had. I verified the cited sites and want to add the family context and a refinement to your fix ordering.

1. Verified

• session-persistence-jsonl/src/index.ts:422-427 (appendBatch → appendLines) appends blindly with no durable seq-continuity check — confirmed.
• format.ts:362-368 (SessionLogScanner) throws seq gap in committed region exactly as you cite; the turn/end-present condition at :370 (my line read: if any decoded event is turn/end, throw immediately) is why your case can't self-heal — confirmed.
• The re-emission shape (tool/result → step/end → step/start → assistant/chunk) matches an agent loop resuming an interrupted step from a stale session view.

2. Family context: this is the same-process sibling of the #1333 cross-process seq thesis

On #1333/#1452 I established that seq = log.length (core/session/src/index.ts:629) with no cross-process lock — two live processes can each believe their in-memory length and append the same seqs. Your case is the same root cause in one process: the resume path seeded its seq counter from a stale view (missing the last 4 committed events), then appended those 4 with fresh timestamps. Both variants are one defect: the seq counter is derived from a view that can be stale, and the durable append path never re-validates against the committed prefix. (This is also why wellorbetter's writer-lease patch on #1550 — the community's first fix — addresses the cross-process half but not the stale-view half.)

3. Refinement to your fix ordering

Your write-side invariant (1) is right, but make it cheap and strict: appendLines should compare the batch's first seq against the scanner's committed length before encoding, and refuse loudly (throw) rather than truncate-and-repair — silent truncation at write time can drop events the in-memory view hasn't flushed yet (the same reason I argued against auto-rewrite on the #1518/#1593 family: detect+report, explicit repair, never implicit). The resume-path fix (2) — always derive the counter from a fresh committed-prefix load — is actually the higher-value half: it prevents the stale view from ever existing. Optional reader self-heal (3) is fine as a compatibility nicety, but only after (1)+(2) land.

4. One addition to your repair playbook

Your manual repair is exactly right, and it's worth noting this is the 6th distinct corruption mechanism in the family now: #1333 (cross-process seq duplication) / #1452 (same) / #1497 (torn tail) / #1473 (append vs read race) / #1586 (recovery-writer vs legacy-writer collision) / #2167 (same-process stale-view re-append). The common thread across all six: the append path has no durable ownership/validation contract. A unified hardening — append watermark + durable seq assertion + writer lease (wellorbetter) — would close the whole family, and I've tracked exactly this as a PR-ready workstream (#1333/#1452/#1497/#1473/#1586, now + #2167).

Thanks for the precise evidence — this makes the stale-view trigger a first-class member of the family rather than a one-off.

[KirschBluteX]

I reproduced the corruption mechanism and implemented a candidate fix against 47f9438.

Root cause

The concrete failure is below AgentLoop replay: PersistenceCoordinator checks a batch against its in-memory cursor, but the backend mutation did not prove that the cursor still described the current durable prefix. A coordinator can load or prepare revision A, another coordinator can commit revision B, and the stale coordinator can then pass its local contiguous-seq check and append events from A. That reproduces the four-events-behind signature with fresh timestamps while leaving the file unreadable.

The resume/HMR path can expose the stale coordinator, but scanning the durable event count immediately before append is not sufficient by itself: another writer can commit between that scan and the write. The comparison and mutation need one writer transaction.

Fix

• SessionState now retains the exact source-qualified revision that established its cursor. Append and repair pass that expected revision to the backend; a mismatch rejects without changing storage.
• JSONL uses a root-level withFileLock key derived from the session id, so cwd and compression choices cannot bypass serialization. Existing append/repair compare the stat-derived revision while holding that lock, and first materialization repeats global id discovery under the same lock.
• SQLite uses BEGIN IMMEDIATE; row revision comparison, event mutation, and revision increment occur in the same transaction.
• Cold repair and live/HMR adoption reload after losing the repair CAS instead of publishing the stale source.

Additional findings fixed while exercising the race

• A successful JSONL rollback changes the file's stat revision even when it restores the same logical prefix. The backend now returns that recovered revision to the coordinator before rethrowing the original append/fsync error, so the unchanged batch remains retryable.
• The JSONL lock must be keyed above cwd and compression; artifact-path locking still permits duplicate global session ids.
• Live adoption must reread and revalidate the seed after another writer wins torn-tail repair; retaining the rejected initialization promise leaves the live session permanently stuck.
• The Python SDK runtime dependency closure now includes @deepseek-ai/dsh-atomic-write, which JSONL reaches at runtime.

Regression evidence

• Two coordinators starting from one JSONL revision: exactly one append commits in both raw and zstd modes.
• A writer exactly four events behind rejects repeatedly and leaves the durable log contiguous and unchanged.
• Stale JSONL/SQLite repair cannot replace a newer committed closer.
• First materialization across different cwd/compression choices publishes exactly one artifact.
• Live adoption reloads after a competing repair; append/fsync rollback can retry without a seq gap; SQLite mid-batch failure rolls back transactionally.
• Related suite: 52 files, 1,454 passed, 1 skipped. Focused final rerun: 17 passed. Host/client build, typecheck, lint, and package/runtime-closure checks passed for the affected surfaces.

Implementation: commit 08415957 · one-commit compare against master