corrupt session log: seq gap in committed region after restart when session/end-seed was committed in a separate append

[wpc0323]

corrupt session log: "seq gap in committed region" after restart when session/end-seed was committed in a separate append

Symptom

After restarting DSH, opening a previously-used session fails with:

history unavailable for session "session-XXXX": Error: corrupt session log: seq gap in committed region at line 5 (expected 4, got 3)

and resuming the same session for chatting fails with:

internal: resume failed for session "session-XXXX": Error: corrupt session log: seq gap in committed region at line 5 (expected 4, got 3)

Two sessions were affected on this machine. Both had been created and used in a previous DSH run, then became unreadable on the next startup. verify-logs-style inspection confirmed the corrupted pattern in both.

Observed log state

In the affected session.jsonl.zstd, the committed region contained two events with the same seq:

seq 0  permission/preset
seq 1  sandbox/mode
seq 2  approval/policy
seq 3  session/end-seed        <- committed in its own append batch
seq 3  agent/inbox/spliced     <- next write reused seq 3 (duplicate)
seq 4  ...

The validator walks the committed prefix and requires contiguous seq = 0,1,2,...; it reports expected 4, got 3 at the duplicate line. Because the duplicate breaks contiguity, the committed region is truncated there and every later event (the whole conversation) becomes unreadable — the session's history is effectively lost from the readable region.

Suspected root cause

Both writers computed the same next-seq from the same committed base (3 events: seq 0..2), but the session/end-seed was written by a path whose commit was not included in the base the next writer used:

1. When DSH creates or resumes a session whose seed does not already end with session/end-seed, the Session constructor appends one:

   // dsh-session, Session constructor
   this.firstLiveSeq = this.log.length;
   if (seed !== void 0 && this.log.at(-1)?.type !== "session/end-seed")
     this.append("session/end-seed", {});

   This appended session/end-seed is a live-session write, flushed to persistence as its own append batch (it occupies a seq that a consumer reading stored history must account for).

2. On the following startup, the recovery/next-seq computation for that session apparently did not count that separately-committed session/end-seed batch, so the first post-resume write (agent/inbox/spliced) reused the end-seed's seq → duplicate seq in the committed region.

This is presented as the suspected mechanism from the observed log evidence and the constructor behavior above; I could not fully trace the recovery-cursor code path, so the exact line responsible still needs confirmation. The bug is in DSH's own session lifecycle/persistence (the affected sessions were created by DSH itself, not by any external tool).

Reproduction

The trigger appears to be: a session whose log received a session/end-seed in a separate append batch (the live-session append path described above), followed by a DSH restart, followed by a new write to that session. It did not reproduce for sessions whose whole seed (including any end-seed) was committed in a single append.

Impact

• Session history becomes unreadable after restart.
• Resume fails (cannot prepare/corrupt log), so the session cannot be selected/used.
• Events after the duplicate seq are dropped from the readable committed region (history truncation).

Workaround

Manually repair the log by removing the duplicate line (the separately-committed session/end-seed whose seq was reused), restoring contiguous seqs, and re-encoding the zstd frame. This recovered both affected sessions successfully. Take a backup of ~/.dsh/sessions/**/session.jsonl.zstd first.

Environment

• DSH version: 0.1.0-rc.6
• Platform: Windows (schannel/TLS sandbox environment)

[MedicineKing]

Same family as #1497 — seq overlap/rollback in the committed region. Two zero-history-loss options (verified against the unpatched npm 0.1.1-rc.2: 931,455 events, no seq gap, Session.fromRestore passes):

1. Read-side tolerance patch: SessionLogScanner skips replayed committed rows (seq < expected) instead of refusing the whole log — everything already on disk stays, nothing gets cut. Your exact shape — end-seed committed in a separate append, restart resumes after it — is the recovery-pair the rollback rule targets; the duplicate end-seed row is what gets skipped.
2. Zero-install CLI (dsh-corrupt-session-repair): node dsh-corrupt-session-repair.mjs scans (read-only) → fix previews → fix --apply (.bak.zstd first; writes official checksummed frames — header line alone in frame 1, so assertZstdHeaderFrame accepts it). Real seq holes / physical corruption are reported, never rewritten.

Pick #2 if the log is already broken; patch the readers if you want future crashes to stay invisible.

[Tazio7]

Three independent reproductions of the resume-after-end-seed seq collision (macOS, dsh 0.1.1-rc.2) — plus a secondary publication failure after seq repair

Following up on this thread and #1364 with three incidents from a single macOS deployment today (two daemons, four affected session logs). Our evidence adds three points not yet covered here: (1) the collision is independent of write timing, (2) seq repair alone is insufficient — a secondary publication failure persists until the session identity is rotated, and (3) a workable containment design.

Environment

• dsh 0.1.1-rc.2 (@deepseek-ai/dsh, @deepseek-ai/dsh-session, @deepseek-ai/dsh-session-persistence, @deepseek-ai/dsh-session-persistence-jsonl all 0.1.1-rc.2)
• macOS 26.6.2 arm64, Node v26.5.0; daemons managed by launchd
• Sessions: zstd multi-frame JSONL (default compression); two long-lived lark-channel daemons, each serving a private chat and a group chat

Incident chain (all three logs, identical signature)

Each incident started with a daemon restart while a conversation had a live agent:

1. The retiring process drained and appended session/end-seed (as its own frame).
2. The replacement process, on the next message, resumed with a stale base and emitted the next agent/inbox/spliced reusing the end-seed's seq.
3. Every subsequent load of that log failed:

corrupt session log: seq gap in committed region at line 108 (expected 178, got 177)

Observed on three logs: advisor group chat (seed seq 177), juna private gen-1 (seed seq 78), juna private gen-2 live session (seed seq 18562).

Evidence ① — collision is independent of write timing

Time gap between the drained end-seed and the colliding write: 42 seconds in one incident, ~1 hour in another. The colliding event is written long after all I/O has settled, so this is not a torn-write/flush race — the resumed process deterministically computes its next seq excluding the end-seed the previous lifecycle persisted. In the 1h case the intermediate turns were fully functional; only the accounting was wrong.

Evidence ② — seq repair alone is insufficient: secondary alias rejection on publish

We repaired the log surgically: removed the duplicate agent/inbox/spliced line, validated byte-level with decodeStorageRecord from @deepseek-ai/dsh-session (the same decoder the loader uses — our validator reproduced the exact error text and numbers on the corrupt file before repair). Seq contiguity was restored, multi-frame layout preserved.

After repair, resume still failed — now at a later stage:

cannot publish session "lark-oc_…": persisted state already owns this identity

i.e. the alias rejection in SessionPreparations.reservationFor. It persisted across daemon restarts (fresh preparation pool) and was only cleared by rotating the conversation to a fresh session identity (epoch bump via the /new mechanism). Reading dsh-session-persistence, this is reach() catching the resume failure and create() publishing a second Session for an id whose earlier preparation is still reserved.

Upshot: when a log ends in [end-seed → events appended by the stale-base process], removing the duplicate line is not enough — the whole post-seed segment is poisoned and the durable fix is to truncate at the end-seed (validated live on one log; a second log was recovered by epoch rotation instead).

Evidence ③ — containment design that survived contact

We deployed a session-log doctor (exit-code contract for launchd/heartbeat integration):

• Validation = loader semantics: per-frame decode, first frame must be exactly the header line, per-event seq contiguity via decodeStorageRecord (chunk rows expanded), torn final frame/line tolerated as BUSY (skip, don't false-positive — we initially flagged mid-write scans and got alert noise).
• Repair policies, all frame-preserving (only affected frames re-compressed; untouched frames byte-identical):
  • REFRAME — first frame not exactly the header line → rebuild as header-frame + body-frame (single-frame recompression also trips the header assertion — one of our own bad repairs produced exactly this);
  • DUP_SEED — end-seed directly followed by a stale-seq line → truncate at the end-seed (drops the poisoned segment, keeps full history before it);
  • DUP_ONLY — other stale-seq rows → drop the rows.
• Backups are stored outside the sessions tree.
• Wired into: a mutual-restart helper, the scheduler's rescue executor, and a 60s heartbeat check (auto-repair; unrepaired damage escalates to a chat alert).

Regression-tested against four real corrupt logs from today: all detected, repaired, and re-validated CLEAN.

Questions

1. Can you confirm the mechanism: the retire path persists session/end-seed into the durable log, but resume-side reconstruction excludes boundary markers from the logical event stream — so the first post-restart append reuses the seed's seq?
2. The master docs (session.zh.md) describe locating the last session/end-seed in stored history rather than assuming one at firstLiveSeq — is that the intended fix, and which release will carry it?
3. For already-affected logs: is truncate-at-end-seed sanctioned, or does it break downstream assumptions (e.g., compaction entry state referencing pre-seed seqs)?

Happy to provide the affected logs (redacted) or the validator/repair script if useful.

[Tazio7]

Follow-up: this hit three more times today on a second macOS deployment (dsh 0.1.1-rc.2, launchd-managed daemons, zstd multi-frame JSONL sessions) — identical signature each time. After a restart, the resumed process re-emits agent/inbox/spliced with the seq of the session/end-seed written by the retiring process's drain. Gaps between the drained seed and the colliding write were 42s and ~1h across incidents, so this is not a flush/timing race — the resumed base deterministically excludes the seed.

Two additions to the evidence base:

1. Seq repair alone is not enough — secondary publication failure.
After restoring seq contiguity (surgical removal of the duplicate line, validated byte-for-byte with decodeStorageRecord — our validator reproduced the exact loader error text before repair), agents.resume still failed at a later stage:

cannot publish session "…": persisted state already owns this identity

— the alias rejection in SessionPreparations.reservationFor, persisting across daemon restarts (fresh preparation pool). Service was only restored by rotating the conversation to a fresh session identity (chatEpochs bump, the /new path). The whole post-seed segment written by the stale-base process appears poisoned for publication even after the duplicate line is removed — truncate-at-end-seed (or rotation), not dedup, is the working remedy.

2. Containment that works (until the fix ships).
A deterministic session-log doctor — loader-semantics validation via decodeStorageRecord (header-frame assertion + per-event seq contiguity, chunk rows expanded, torn final frame tolerated), frame-preserving truncate-at-end-seed repair, external backups — wired into a 60s heartbeat and our restart helper. It self-healed one live recurrence within a single scan cycle. Zero model calls, ~1s per full scan.

Question on the fix: is the resume-base accounting (the resumed lifecycle's next seq counting the drained session/end-seed) already addressed on the 0.1.2 line — e.g., by the Session.events → on-demand read refactor (seq/eventAt()/snapshotEvents()) and the isSeeded / inheritedEventCount boundary accounting present in 0.1.2-alpha.4 (absent in 0.1.1-rc.2)? If yes, which release would be safe for a production deployment? Happy to validate a candidate build against our three affected logs (redacted copies on hand).

Environment: dsh 0.1.1-rc.2 full stack, macOS 26.6.2 arm64, Node v26.5.0.

[xdongHo]

Independent confirmation from a third deployment (Windows 11, dsh 0.1.0-rc.6, zstd JSONL store) — this thread's mechanism reproduced twice for us, byte-for-byte:

• session/end-seed at seq 3 persisted in its own frame; 3.4 h later the resumed writer re-appended agent/inbox/spliced at seq 3 and continued contiguously (40,419 further events).
• Same shape at scale: session/end-seed at seq 354,320 (one frame), resumed agent/inbox/spliced at seq 354,320, continuing to 378,636.

We wrote up the full three-layer root cause with frame-level timelines, a validated fix, and pointers to this thread and #2167 in #5408: #5408

TL;DR of the fix we run locally (verified on all three corrupted sessions + a 23-case integration suite over zstd and plain layouts): a seq-aware convergence step in appendLines before the append handle opens — decode the log tail from its bytes (frame granularity for zstd), classify the batch's first seq: clean continuation passes through untouched (3.6 ms fast path on a 4.1 MB / 16k-frame log); a superseded tail (this thread's end-seed, or the interrupted-close block) is truncated at the frame boundary below the branch point and the writer's continuation lands contiguously — the corruption becomes self-healing on the first post-resume append; durable gaps / row-straddling boundaries refuse loudly. Happy to share the patch.