session-persistence-jsonl: recover from stale duplicates left by an interrupted rewrite (seq gap makes history unreadable)

[zhangyuyao-zx]

Bug: seq gap in committed region makes session history unreadable after an interrupted rewrite

Symptom

The web GUI shows: history unavailable for session "...": Error: corrupt session log: seq gap in committed region at line N (expected 87804, got 87800) (internal).

On-disk, ~/.dsh/sessions/<project>/<session>/session.jsonl.zstd contains stale duplicate rows: when the dsh web process is killed/restarted while writing (or two writers race during a restart), an interrupted rewrite leaves an older segment re-emitted after the cursor already passed it. The row at the gap contains events with seq < events.length, so SessionLogScanner.consumeEventLine (packages/session/session-persistence-jsonl/src/format.ts) sets issue and — because the duplicated tail includes turn/end rows — throws. The whole session then fails to load every time, even though the committed prefix and the post-duplicate tail are both intact.

Also observed in the same restart pattern: corrupt Zstandard session log: first frame is not exactly one header line (torn first frame), which is a separate code path.

Why this is recoverable

The duplicated rows are byte-identical re-emissions of events the scanner already accepted (their seqs are strictly below the cursor). The file is contiguous except for that duplicated region: skipping stale events until the cursor resyncs reconstructs the exact original event stream. Genuine data loss (a forward gap, event.seq > events.length) is not recoverable this way and should stay fatal.

Proposed fix (read-side only, ~10 lines)

In consumeEventLine, distinguish the two mismatch directions:

const rowStart = this.events.length
for (const event of decoded) {
  if (event.seq === this.events.length) {
    this.events.push(event)
    continue
  }
  if (event.seq < this.events.length) {
    // Stale duplicate left by an interrupted rewrite: skip it and keep
    // scanning; the cursor resyncs once the duplicated region ends.
    continue
  }
  // A forward gap (events genuinely missing) is unrecoverable.
  const expected = this.events.length
  this.events.length = rowStart
  this.issue = new Error(
    `corrupt session log: seq gap in committed region at line ${this.eventLine} `
    + `(expected ${expected}, got ${event.seq})`,
  )
  if (decoded.some(candidate => candidate.type === 'turn/end')) throw this.issue
  return
}
this.committedBytes = endByte

Behavior change: a duplicated turn/end tail no longer throws; it is skipped and the read completes with the deduplicated stream. Forward gaps still throw with the original message. Unparsable rows keep their existing semantics.

Tests

Two cases added to packages/session/session-persistence-jsonl/tests/jsonl.spec.ts:

• skips stale duplicate events left by an interrupted rewrite and resyncs — duplicates spliced mid-log; expects the original event array and full committedBytes.
• skips a stale duplicated suffix even when it repeats turn end events — duplicates at the tail including turn/end; previously this threw, now it reads cleanly.

End-to-end verified against the real readPrefix path with synthetic zstd logs (header frame + event frames), including the exact "duplicate tail with turn/end" shape from the field report, plus a forward-gap control that still fails as expected.

Standalone patch

A ready-to-apply patch (plus diagnostic/verification scripts) is published here:
https://github.com/zhangyuyao-zx/deepseek-harness-mac/tree/main/session-repair

Happy to adapt it to whatever direction the team prefers (e.g. truncate-at-gap instead of skip-and-resync, or treating the whole post-gap tail as uncommitted).

---

（中文摘要）dsh web 进程在写入中被中断/重启时，会话日志会残留重复行，导致 seq gap in committed region 报错、历史无法加载。建议在扫描器里把 seq < 游标 的过期重复行改为跳过并对齐，真正的前向缺口保持致命错误。附测试用例与可直接应用的补丁。