Bug: two different session events share the same seq — "corrupt session log: seq gap" on load

[hangmujun]

Bug: two different session events share the same seq — "corrupt session log: seq gap" on load

Environment

• DeepSeek Harness @deepseek-ai/dsh 0.1.0-rc.6 (web profile, Windows 11)
• Node 22.23.1
• Session persistence: session.jsonl.zstd (zstd, packed chunk rows)

Symptom

Loading a session fails with:

history unavailable for session "session-ad228b75-13e8-4979-bfbd-33d84a9ddb27":
Error: corrupt session log: seq gap in committed region at line 30976
(expected 461421, got 461420)（internal）

Evidence (from the damaged artifact)

Decompressing the log and inspecting around the reported line shows two different events that were both allocated seq 461420:

{"type":"session/end-seed","seq":461420,...}
{"type":"agent/inbox/spliced","seq":461420,"time":1786716691871,
 "data":{"target":"next-turn","start":0,"inserted":[{...user message...}]}}
{"type":"turn/start","seq":461421,...}

The loader (SessionLogScanner.consumeEventLine) requires event.seq === events.length (0-based expanded-event index). Because agent/inbox/spliced carries the same seq as the preceding session/end-seed, the check fails and the whole log is refused.

All 471,075 expanded events before and after this point are strictly contiguous (verified by replaying the same decodeStorageRecord expansion the loader uses), so this is the single corruption in the file.

Root-cause hypothesis

The two events come from different write paths:

• session/end-seed — appended by the Session constructor (seed boundary on reopen/resume),
• agent/inbox/spliced — appended by the agent loop when a user message arrives.

Both received seq 461420, and the counter continued correctly at 461421 afterwards. This looks like a race in seq allocation across concurrent appends (e.g., the next-seq value was derived from a stale log-length snapshot, or the append path is not serialized across these two writers). Since the two lines carry different event types, this is not a torn-write duplicate of one event — the allocator handed out the same sequence number twice.

Impact

A session with one duplicated seq becomes entirely unloadable (the loader refuses the whole log, not just the bad event). The user loses access to the whole conversation history until manual repair.

Suggested direction

• Serialize seq allocation per session (single monotonic counter updated atomically with the append), or
• Have the loader tolerate/skip a single duplicate-seq row (log a warning and continue) instead of failing the whole log, or
• Investigate the session/end-seed vs agent/inbox/spliced interleaving around session reopen while a message is queued.

Notes

• We repaired the affected file locally by removing the duplicate agent/inbox/spliced row and rebuilding the zstd frames (frame 1 = header line, frame 2 = events); the session loads fine afterwards. The paired inbox-removal row and the user/message for the same input were both intact, so no model-visible history was lost.
• I could not reproduce this deterministically — it happened once under normal use (message sent while the session was being reopened/restored).

[zoahdev]

Thanks for decompressing and pinning the exact line - that is the smoking gun.

Reading

SessionLogScanner.consumeEventLine requires event.seq === events.length, so a single duplicated seq (two events both carrying 461420) makes the next expected index disagree with the stored seq and every subsequent load fails as seq gap. The real bug is upstream: seq minting assigned the same number to session/end-seed and agent/inbox/spliced in the same turn.

Likely cause to verify

Two writers both incremented from the same observed lastSeq without a single atomic owner for the seq counter - plausible when the turn teardown path (session/end-seed) and the inbox/splice path both append in the same tick. A fix would be to make the session seq monotonic allocation a single critical section (or assign the seed at append time through the same allocator every event already uses).

Recovery today

The content is intact - export it with dsh-shelf (works even when the session won't load):

npx dsh-shelf rescue <id>

Happy to prepare a cherry-pick-ready patch once I can reproduce the exact interleaving (a captured session dir + the events around line 30976 would let me pin the two writers).

[zoahdev]

Shipped the defensive half while the exact interleaving still can't be reproduced deterministically.

Why it isn't a mid-append race

Session.append is synchronous and guarded against re-entrancy (attachments.get(this).appending), and the persistence coordinator serializes appends per session (serialize(id) around appendCore) while asserting event.seq === state.cursor + i. So two appends through one coordinator cannot mint the same seq. The duplicate almost certainly comes from the reopen/restore boundary: a restore appends session/end-seed (seq = seed length) while the live loop appends agent/inbox/spliced (same seq), and both reach the same file.

Shipped: tolerate one duplicate on load

Branch: zoahdev/deepseek-harness@fix/session-log-duplicate-seq-tolerance (cherry-pick-ready against 47f943859b).

• packages/session/session-persistence-jsonl/src/format.ts: an event whose seq equals the last accepted event's seq is skipped and recorded in skippedDuplicateSeqs; the next event still validates against the contiguity contract. A real gap (seq too high) and out-of-order corruption still fail.
• Regression test added; session-persistence-jsonl jsonl.spec 152/152 green.

This is the "don't lose the whole history over one duplicated row" fix. The durable fix — serializing the restore/adopt boundary so two logical writers can't coexist — still needs your captured session dir + events around line 30976 to pin the exact writer pair. The sqlite backend has the same committed-region check; I'll mirror the tolerance there too if maintainers want parity.

[zoahdev]

sqlite parity is done too: zoahdev/deepseek-harness@fix/session-log-duplicate-seq-tolerance-sqlite (cherry-pick-ready against 47f943859b). Same one-duplicate tolerance, with the expected-seq cursor tracked separately from the row index so a skipped duplicate isn't misread as a torn tail. scanRows regression added; 9/9 scanRows tests green.

The durable fix (serializing the reopen/adopt boundary so the two writers can't coexist) still needs the repro sample. Both backends now survive the symptom regardless.

[boyin111-1]

Thanks for the detailed write-up — the pinned seq collision is exactly the failure mode our offline health checker now catches, so I wanted to share a ready-made diagnostic for it.

dsh-doctor --session <log> performs the same contiguity scan the loader runs (SessionLogScanner.consumeEventLine: event.seq === events.length) over the expanded event stream — since rc.6, consecutive assistant/chunk deltas are packed into single storage rows (text-chunks / reasoning-chunks / tool-call-chunks, chunk-rows.js decodeStorageRecord), so the check expands each packed row to its seq0+k sequence before comparing, and malformed packed rows fail loud exactly like the loader. Against your damaged artifact it would report the duplicate at the exact line:

✗ 会话 seq 倒退: <log> — L30976 seq 461420 倒退（已见 461421）（#1433/#1452/#1586 并发写/重放导致 seq 撞号）

i.e. it distinguishes the three shapes you describe — gaps (expected→got), duplicates (a seq already seen), and rewinds — with line numbers + expected/got, which is what you needed to hand-repair the file. It's zero-dependency, runs before boot, and is safe to run against a live log (read-only).

Two notes for the thread:

1. Your duplicate is the "rewind" shape (two events both carrying 461420 after …419), which the checker flags distinctly from a gap — so it's a quick way to tell "allocator handed out the same seq twice" from "a run was dropped".
2. It doubles as a regression check for the fix: once the tolerance/serialization lands, re-running --session on previously-broken artifacts should flip from ✗ seq 倒退 to ✓ seq 连续 (modulo the tolerated duplicate), which is a cheap way to verify the repair on real logs without reimplementing the loader.

Happy to open a PR or a discussion over in boyin111-1/dsh-doctor if the maintainers want the expansion semantics or the fixture pairs (packed-row healthy log, malformed packed row, duplicate-seq) as a reference.