[Bug] Two concurrent owners corrupt one session log, then the whole history fails to load with a misleading "torn JSONL record" error

[Van1145141919810]

Summary

A session that is still live in one DSH process can be resumed by a second process. The second
process performs a durable crash repair of the open tail turn (synthetic interrupted-turn closers
appended to the log) while the first process is blocked in a long tool call. When that tool call
returns, the first process appends its real result starting at the same seq using its own
in-memory cursor. The log then contains two interleaved, seq-overlapping histories, and every later
load of that session fails — the entire history, not just the tail — reporting:

corrupt Zstandard session log: complete frame contains a torn JSONL record

which is misleading: every Zstandard frame is checksum-valid and every JSONL record parses. The real
defect is a seq regression, and the exact reason is recorded in SessionLogScanner.issue but
discarded by the throw at the end of readZstdPrefix.

Environment

• @deepseek-ai/dsh 0.1.2-rc.1 (also dsh-session-persistence 0.1.2-rc.1, dsh-session-persistence-jsonl 0.1.2-rc.1, dsh-session 0.1.2-rc.1)
• Node v24.15.0, Windows 11, JSONL backend with compression: "zstd"
• Observed on two independent sessions of one installation (34 session logs scanned, 2 affected)

Mechanism (what actually happens)

1. Process A owns session S. It writes tool/call (seq N) for a blocking tool
   ({job_id, wait: true, timeout_ms: 120000}) and blocks for two minutes. A's in-memory cursor is
   N+1.
2. Process B resumes the same session id while A is still blocked. B reads the log, sees an open
   turn with a recorded-but-unresolved tool call, and treats it as crash residue:
   interruptedTurnClosers() produces synthetic events (an error tool/result
interrupted-tool-result-<callId>-<seq>, step/end, turn/end{reason:"interrupted"},
   session/end-seed), and they are written into the same file — sequences N…N+3. B's cursor
   becomes N+4, and B starts its own new turn.
3. A's tool call returns and A appends the real tool/result at seq N — its own stale cursor —
   and continues its turn. A never notices, because the live-writer path appends from in-memory
   state.cursor without re-checking the file revision.
4. Both processes keep appending concurrently. The resulting file is a valid sequence of frames, but
   its event chain is not monotonic: it reaches some seq M, then jumps back to N.
5. Any later load fails the gapless-chain requirement and rejects the whole session.

Evidence (seq-level, from the two real logs)

Log 1 (larger, 2.8 MB, 5,845 frames, 10,122 records, all frames checksum-valid, last byte newline):

position	content
seq N (e.g. 190214)	process A: tool/call → blocking tool {job_id:"<job>", wait:true, timeout_ms:120000}
seq N+1…N+4 (same second)	process B: synthetic closers interrupted-tool-result-<callId>-<seq>, step/end, turn/end{reason:"interrupted"}, session/end-seed — written durably by B
~2 min later	process A: the real tool/result at seq N+1 (same callId, real output), turn continues
then	59 alternating segments belonging to A's turn and B's turn; seq region N+1…M written twice, seq N+1 appears twice with different payloads
break point	the loader stops at M_committed, then sees N+1 → chain broken at byte 6,955,318 (JSONL line 9853)

Log 2 (smaller, 349 KB): same mechanism, only 3 divergent events, which makes it easy to read:

• seq 3819: synthetic interrupted-tool-result-<callId>-3819 (process B) vs the real tool/result (process A)
• seq 3820: step/end in both branches
• seq 3821: turn/end{reason:"interrupted"} (B) vs step/start{step:8} (A)
• A's branch continues to seq 24814; B's branch contributed only those 3 synthetic events

Reconstructed event counts using the real backend (readPrefix): branch that ends at the newest
write = 193,806 events (log 1) and 24,815 events (log 2); the other branch of log 1 = 193,179 events.

Why the error message is misleading

SessionLogScanner.consumeEventLine() records the real cause in this.issue
(corrupt session log: unparsable committed event at line L or
corrupt session log: seq gap in committed region at line L (expected X, got Y)) and only throws it
immediately if a later record in the same scan is a turn/end. Otherwise the scan ends and
readZstdPrefix throws the generic byte-count message, discarding issue:

// packages/session/session-persistence-jsonl/src/index.ts — readZstdPrefix
const complete = scanner.checkpoint();
if (complete.committedBytes !== complete.inputBytes) throw new Error("corrupt Zstandard session log: complete frame contains a torn JSONL record");

committedBytes !== inputBytes is not sufficient for "torn record": a torn record can only occur in
an incomplete final frame, and this code path handles complete frames only. That is why the same
defect produced two different messages in two logs — the log whose divergent branch happened to
contain a later turn/end reported the truthful cause, the other one reported "torn JSONL record".

Code locations (0.1.2-rc.1)

• packages/session/session-persistence/src/index.ts — prepareCore() (interruptedTurnClosers(storedEvents)),
  commitPrepared() → backend.commitRepair(...), borrowSession() (read-only observation, no commit),
  appendCore() → adopt() → prepareCore() + commitPrepared(), appendCore()'s
  state.cursor-based seq check.
• packages/session/session-persistence-jsonl/src/index.ts — commitRepair(), SessionLogScanner
  (issue, checkpoint()), readZstdPrefix(), scanZstdFrames().
• packages/session/session/src/repair.ts — interruptedTurnClosers() (timestamps reuse the last real event).
• The only ownership guards are per-process in-memory: already has a live persistence owner,
  cannot prepare session "..." while it is live, and the API session controller's
  hasApiSessionSubagentOwner. There is no cross-process lock or claim on the log file.

Suggested fixes

1. Cross-process ownership: take an exclusive claim on the session log (lock file /
   LockFileEx) before commitRepair/appendBatch; or verify that the durable revision has not
   advanced since the in-memory state was built, and fail loudly instead of interleaving.
2. Do not durably write repair closers into a log that still has a live writer. borrowSession
   already shows the read-only pattern (repair in memory, no commit); a cold load could do the
   same, or write repair artifacts to a side file.
3. Surface scanner.issue in readZstdPrefix (and only use the "torn record" wording when a
   record really lacks a terminator), including the byte offset of the break.
4. Optional salvage mode: allow loading the maximal committed prefix with an explicit warning,
   since committedBytes is already computed. Please do not auto-truncate: on a seq regression
   the other branch's data would be silently discarded, so the user must choose.

A one-line equivalent of fix 3 that I applied locally to diagnose this:

const issue = scanner.issue;
const detail = issue === void 0 ? `complete frame contains a torn JSONL record at byte ${complete.committedBytes}` : issue.message.replace(/^corrupt session log: /, "");
throw new Error(`corrupt Zstandard session log: ${detail}`, issue === void 0 ? void 0 : { cause: issue });

Workaround for users

• Only run a given session in one place at a time. Viewing history is read-only and safe; the
  dangerous action is resuming/continuing a session id that another process still runs
  (a second terminal, a second browser tab/window, GUI + CLI).
• The dangerous window is "turn open with an in-flight tool call" (blocking job_output{wait:true},
  long-running shell tools, waiting on approvals).
• If it happens: back up the log, then rebuild a gapless chain (the maximal contiguous chain ending
  at the newest write) and preserve the other branch as its own session. Both branches were fully
  recoverable in my case.

Recovery data point

Both affected sessions were recovered with the real readPrefix verifying the rebuilt logs:
193,806 events (torn=none), 193,179 events (torn=none), 24,815 events (torn=none).

---

I can provide a scrubbed minimal reproduction (no user data) or a synthetic log pair on request.

[PerryLink]

Verified against deepseek-harness master @ c291e7961a (2026-09-10, 0.1.5-rc.2) and tag dsh-v0.1.2-rc.1 (the version this report names). Code references below use those two points unless stated otherwise.

Thank you for this report — the level of detail made it directly verifiable. I checked every claim against the source, and the diagnosis is correct in substance. Summary up front:

• Root cause: confirmed. 0.1.2-rc.1 has no cross-process write exclusion, and the cold-load crash repair (commitRepair) does durably append synthetic closers into the live log. The interleaving mechanism works exactly as you describe.
• Misleading message: confirmed, with one small nuance about the "torn record" wording (below).
• The corruption cause is already fixed on master — by a cross-process write lease that landed the day after dsh-v0.1.2-rc.1 was tagged. Your fix 1 is implemented; fix 2 is subsumed by it. Fix 3 and fix 4 are still open on master.
• A few minor corrections: two of your code paths, one detail of the synthetic closer sequence, and an off-by-one in step 2.

Root cause — confirmed

At dsh-v0.1.2-rc.1:

1. No cross-process lock exists. git grep for flock/LockFileEx/lock files over packages/session at that tag finds nothing. (There was an earlier wx-created .lock sibling, but it was deliberately removed in 270a06b38b "fix(session-persistence-jsonl): drop crash-unsafe cross-process log lock", 2026-08-19, because a stale lock permanently wedged the log and cleanup failures could duplicate seqs.)
2. The only ownership guards are in-process. The coordinator's private states = new Map<SessionId, SessionState>() with its owner?: Session field, commitPrepared's session "${id}" already has a live persistence owner (coordinator.ts ~L1120), prepare()'s cannot prepare session "${id}" while it is live (~L815/826, gated on ctx.sessions.get(id) — this process only), and the API session controller's hasApiSessionSubagentOwner are all per-process memory. You characterized this exactly right.
3. The live-writer path never re-checks the file. appendCore validates contiguity against the in-memory state.cursor (~L784–788) and backend.appendBatch → appendLines opens the path with open(path, 'a') and writes at the physical EOF with no revision re-check (jsonl index.ts ~L693–721). So process A's real tool/result lands at its stale seq exactly as you describe.
4. Cold load commits the repair durably. prepareCore computes interruptedTurnClosers(storedEvents); commitPrepared calls backend.commitRepair(source.inspection, source.tornMarker, source.closers) (~L1123–1124), and the JSONL commitRepair appends the closers into the same file through appendLines (~L469–478). Process B's only freshness protection is the stat-revision round trip (isPreparedSourceCurrent), which cannot see A's future append while A is blocked in a tool call. Treating an open turn as crash residue is correct from B's perspective — and that is precisely the bug: the two processes have no way to arbitrate.

So: two writers, overlapping seq regions, gapless-chain violated, whole-log refusal on every later load. Confirmed.

The misleading message — confirmed (with a nuance)

At 0.1.2-rc.1, SessionLogScanner records issue as either corrupt session log: unparsable committed event at line L or corrupt session log: seq gap in committed region at line L (expected X, got Y) (format.ts ~L424–454), and throws it immediately only if a later successfully-decoded row contains a turn/end; otherwise the scan ends with committedBytes < inputBytes, and readZstdPrefix throws the generic message, discarding issue. Your quoted snippet matches the tag verbatim (jsonl index.ts L400–402):

const complete = scanner.checkpoint()
if (complete.committedBytes !== complete.inputBytes) throw new Error('corrupt Zstandard session log: complete frame contains a torn JSONL record')

One nuance on your parenthetical "a torn record can only occur in an incomplete final frame": a complete frame can also end in a JSONL record missing its newline terminator — that is the genuine case the wording names (it is pinned by the test "rejects a complete frame containing a torn JSONL record" in zstd.spec.ts). The real deficiency is what you identified: committedBytes !== inputBytes conflates that genuine case with committed-region damage (unparsable rows / seq regression), and the message only names the former. The two-different-messages observation follows mechanically from the turn/end-gated throw — consistent.

Fix status on master (HEAD c291e7961a)

1. Cross-process ownership — implemented. c58097a826 "feat(session-persistence-jsonl): cross-process write-ownership lease" (merged 2026-09-04, Refs #3245) — the day after dsh-v0.1.2-rc.1 (2026-09-03). Write handles hold a kernel-arbitrated lock for their whole lifetime: non-blocking flock(2) on session.lock on POSIX (native addon since d927cbff99, 2026-09-08, shipping in the 0.1.5 line; 0.1.3 used fs-ext), and a named kernel semaphore on Windows. The kernel releases it on process death with no waiting period; there is deliberately no expiry, so a live-but-wedged holder blocks a successor rather than risking exactly this interleaving. Contention maps to SessionAlreadyOwnedError (session "X" is already owned by an active write handle). Read handles never touch the lock. The resume path opens 'write' first (agent-loop ~L879–880), so your scenario now fails loudly at open instead of corrupting. Cross-process behavior is pinned by lease.spec.ts and a real two-process e2e (lease.two-process.e2e.ts, including SIGKILL takeover). This ships from dsh-v0.1.3-alpha.1 onward; please retest the reproduction on 0.1.5-rc.2 — you should now get SessionAlreadyOwnedError for the second owner.
2. Don't durably write closers into a live log — subsumed. Repair closers are appended through the exclusive write handle, which cannot exist while another process owns the session; read-only paths (borrowSession in 0.1.2, read handles on master) never commit repair. A side-file artifact is no longer needed for safety.
3. Surface scanner.issue — NOT done on master. At HEAD, readZstdPrefix still throws the same generic message discarding issue (session-persistence-jsonl/src/index.ts ~L929–932). Your point stands: any log with committed-region damage not followed by a turn/end still reports "torn JSONL record" — including the two logs already corrupted by affected 0.1.2-rc.1 builds, which master will refuse the same way. Implementation note: issue is private on SessionLogScanner in both versions, so your one-liner needs an accessor (or have checkpoint() return it). This is a good first-PR candidate.
4. Salvage mode — NOT implemented. Reads still fail closed with no opt-in prefix recovery, and no auto-truncation exists (agreed it should stay opt-in). Since committedBytes is already computed, an explicit "load committed prefix, warn, keep the raw log untouched" affordance would directly serve recovery of your two logs. Also still open.

Minor corrections to the report

• At 0.1.2-rc.1, prepareCore/commitPrepared/appendCore/borrowSession live in packages/session/session-persistence/src/coordinator.ts (not src/index.ts, which holds the abstract seam), and interruptedTurnClosers is in packages/core/session/src/repair.ts (there is no packages/session/session/src/repair.ts).
• interruptedTurnClosers itself emits only three kinds of closer (tool/result error, step/end, turn/end {interrupted}). The fourth event you observed, session/end-seed, comes from the live seeded Session's prepare (core/session/src/index.ts ~L575–576 at the tag), which B then persists with its first writes. The evidence stays consistent; just note that one event originates elsewhere.
• Step 2 says B's closers occupy seqs N…N+3 while the evidence table says N+1…N+4 — an off-by-one in the prose; the mechanism is unaffected.

Unverified parts

The byte-level evidence from the two real logs (5,845 frames; 193,806 / 193,179 / 24,815 events; break at byte 6,955,318, JSONL line 9,853; 59 alternating segments; 2 of 34 sessions affected) is your artifact data, which I did not reproduce here — though every described observation is fully consistent with the verified code paths. The manual rebuild procedure was likewise not re-run; the "maximal contiguous chain ending at the newest write" property it relies on is exactly what the scanner's committedBytes computes, so it should be sound. A scrubbed repro or synthetic log pair is still welcome and would be ideal input for a regression test alongside fix 3/4 work.

Bottom line: report is accurate; the corruption cause is fixed on master by the cross-process lease (c58097a826, first shipped in v0.1.3-alpha.1, native flock in the 0.1.5 line) — please confirm on 0.1.5-rc.2. The diagnostic surfacing (fix 3) and opt-in salvage (fix 4) remain genuinely open and worth fixing, and the "torn JSONL record" wording still needs the correction you proposed.