Session log can be corrupted when 0.1.0-rc.8 crash recovery races a still-live writer

[iamtouchskyer]

Summary

0.1.0-rc.8 can corrupt a Session log when crash recovery writes synthetic interrupted-turn records while the original writer is still alive. Both writers believe the same seq is next, producing duplicate seq values in the committed region and making the Session permanently unreadable through normal load.

This is a stale-writer / lost write-ownership race, not Zstandard corruption.

Observed production sequence:

seq 1125183  tool/call
seq 1125184  tool/result (synthetic TOOL_OUTCOME_UNKNOWN)  # recovery writer
seq 1125185  step/end
seq 1125186  turn/end { interrupted }
seq 1125187  session/end-seed
seq 1125184  tool/result (real result)                    # original writer
seq 1125185  step/end
seq 1125186  step/start
seq 1125187  assistant/chunk

The next load fails with:

corrupt session log: seq gap in committed region at line 76480
(expected 1125188, got 1125184)

The message says seq gap, but the actual corruption is seq reuse.

Reproduction

1. Persist a tool/call in a Session with no tool/result.
2. Keep the original writer capable of writing, while a second persistence coordinator/backend instance observes the Session as cold. Concurrent history load, resume, backend reload, or a second DSH process can produce this state.
3. The second writer sees an open turn, appends synthetic closers, and resumes the Session with session/end-seed.
4. The original slow tool returns and its writer appends the real tool/result using its old cursor.
5. Both append paths consume the same seq range.

The tool does not need to run twice. A slow live tool call is sufficient.

Root cause in 0.1.0-rc.8

The affected release has separate in-memory cursors with no durable single-writer exclusion:

• appendCore() in session-persistence checks continuity against state.cursor, but that cursor belongs only to one coordinator instance.
• commitRepair() in session-persistence-jsonl appends synthetic closers directly to the JSONL artifact.
• There is no cross-process/write-handle lease preventing the original writer and recovery writer from appending concurrently.

Current status on master

I do not see a new code change to propose here: the current 0.2.1-rc.2 architecture already contains the structural fix.

• SessionWriteLease takes a kernel flock on session.lock for the lifetime of the write handle.
• Write-open acquires that ownership before recovery can append.
• Read handles no longer persist synthetic closers; they return the committed events as stored.
• The current test suite explicitly asserts that a stored open turn is served without synthetic closers and that a second writer cannot open the same Session.

Relevant tests:

• read-only open-turn behavior
• same-root writer exclusion
• two-process contention and crash takeover

Proposed fix

Upgrade the affected runtime to 0.2.1-rc.2 rather than adding a local lock only around commitRepair(). A short repair-only lock still leaves a TOCTOU window between repair completion and resume/write-handle ownership.

The required ownership boundary is:

acquire exclusive lease
  -> read committed prefix
  -> verify durable next seq
  -> append synthetic closers if needed
  -> bind live Session/write handle
  -> append live events
release lease on handle close/process death

If 0.1.x remains supported, this should be treated as a data-corruption fix requiring a backport of the complete write-handle/lease model, not an isolated lock around repair.

Additional context

I repaired the affected local log from a byte-for-byte backup by removing four synthetic interrupted records, preserving the real tool result, and rebuilding the Zstandard frames. The backend then loaded 1,132,924 events from seq 0 through 1132923.

Given the corruption is committed, normal crash-tail truncation cannot repair it. A released-version upgrade notice may be warranted because affected users cannot infer this race from the generic history-unavailable error.

[xiaoshenming]

这个形状（恢复写者先写闭包、原写者用旧游标在同一段 seq 上接着写）我们这边已经能离线修好，并且保住的是真结果、丢掉的是合成闭包 —— 不是反过来。

判据很简单，而且全部在盘上可检：

1. 已提交前缀的末尾正好是官方那一套 interrupted 闭包（interrupted-tool-result-* → 可选 step/end → turn/end{reason.kind:"interrupted"} → 可选 session/end-seed）；
2. 被扫描器挤出来的 overflow 从闭包的第一个 seq 开始、并且连续；
3. 那一段里有真活（tool/call / assistant/message / user/message / …）。

三条都成立时，丢掉闭包、保留 live 写者，seq 一个都不用改 —— 因为两段写的是同一批 seq，本来就对齐；之后尾部若还开着回合并由一个 closer 补上。任何一条不成立就 refuse，不猜。

实测（按你贴的序列构造：tool/call 之后是 recovery 的 4 行，再是原写者的 4 行，seq 从同一点重来）：

• 原文件 → 发布包 readStoredLog 拒（你贴的 seq gap in committed region；我这份小样本因为编码器的帧切分报的是 complete frame contains a torn JSON record，同样打不开）；
• 走一遍修复 → 同一个读端 ACCEPT，日志 10 条事件、seq 连续；
• 保留下来的 tool/result 是原写者的那颗（message.id = r4），合成的 interrupted-tool-result-* 已经不在；再解一次是 ok。

两个边界值得说清：

• 这只能救已经落盘的文件；[bug] 会话历史加载失败：崩溃恢复与残留执行流并发写同一日志，seq 重叠（corrupt session log: seq gap in committed region） #1586 / Bug: corrupt session log after unclean exit — replayed events cause seq gap in committed region, history unrecoverable #1497 / 这条是同一个写侧竞态，再发生一次还是要再修一遍。
• 如果 live 后缀不是从闭包第一个 seq 开始、或者中间不连续（真的丢了字节），我们不会去拼 —— 那种情况下哪一段是真的已经没有判据，report 比“修”更有价值。

工具：xiaoshenming/dsh-session-surgeon。更正一处我原来写错的：inspect 对这个形状报的是 seq-gap-tail（拼接只发生在计划阶段），repair 的计划里才会写明 live-writer-tail 与 stitch 点。repair 默认 dry-run，--apply 先写 .bak.<utc>。

[iamtouchskyer]

Update / correction after checking the released packages: the write-side race itself is already fixed upstream, and the fix predates the current releases.

• SessionWriteLease landed in 0.1.3-alpha.2 (npm published 2026-09-07). I unpacked the published tarballs: 0.1.7-rc.2, 0.2.0-rc.2, 0.2.1-alpha.5, 0.2.1-rc.1/2/3 all contain it; 0.1.0-rc.8, 0.1.1-rc.2 and 0.1.2-rc.1 do not.
• It is a kernel lease, not a lock file: POSIX takes a non-blocking flock(2) on session.lock, Windows holds a named kernel semaphore. It is held for the whole life of the write handle, contention maps to SessionAlreadyOwnedError, and the kernel releases it on process death — so a crashed holder cannot wedge a successor, and a wedged holder is never expropriated.
• Both write paths acquire it: open(access: 'write') acquires before parsing/recovery and keeps it through the recoveredTail flush, and a created session acquires it immediately before its first materializing write.

So on >= 0.1.3-alpha.2 a crash-recovery writer and a live writer can no longer hold the same session at once; the seq-overlap shape in this report belongs to <= 0.1.2. What is not fixed is files those older versions already corrupted: the lease prevents new overlap but cannot repair a torn log, so those still need the offline stitch.

Suggested resolution for this thread: retitle/downgrade to "corrupted by pre-0.1.3 versions — offline repair needed", with the actionable ask being the upgrade path (0.1.7-rc.2 or 0.2.1-rc.3) plus repair of existing files.

[iamtouchskyer]

Closing with the verified outcome.

Root cause fixed upstream. SessionWriteLease (kernel flock(2) on POSIX, named kernel semaphore on Windows) shipped in 0.1.3-alpha.2 (2026-09-07) and is present in published 0.1.7-rc.2, 0.2.0-rc.2 and 0.2.1-rc.1/2/3. Both write paths take it: open(access:'write') before parsing/recovery and through the recoveredTail flush, and a created session before its first materializing write. Crash-recovery and live writers can no longer hold the same session concurrently, and the kernel releases the lease on process death so a crashed holder cannot wedge a successor.

Scope boundary. Versions <= 0.1.2 (0.1.0-rc.8, 0.1.1-rc.2, 0.1.2-rc.1) have no lease and can still produce this seq-overlap shape. The lease prevents new corruption but cannot repair files already damaged, so pre-0.1.3 files still need the offline stitch described above.

Local check. I scanned both DSH session roots on this machine with dsh-session-surgeon: ~/.dsh/sessions (1482 sessions) and the pre-0.2.1-rc.2 backup (902 sessions) — 0 failed frames, 0 corrupt logs. No repair was needed here.

Reopening would be warranted if the same seq-overlap shape is observed on >= 0.1.3-alpha.2, which would mean the lease is not covering some write path.

[Sharkblade2021]

Reopen condition met on 0.2.1-rc.3 (Windows 11 build 26300, Electron 44.7.0, app.asar built 2026-10-29 22:02, installed 2026-10-31 01:45:38 +08).

Three sessions in one workspace, corrupted by this build between 11:41 and 12:25 on 2026-10-31 — complete frames, valid JSON, seq dense 0..N-1 except one re-used block:

session-0382f57c: ...2552 approval/asked | [resume 11:41:16] 2553 interrupted tool/result, 2554 step/end, 2555 turn/end{interrupted}, 2556 session/end-seed | [stale writer 11:48:00] 2553 approval/decided, 2554 job/registered, 2555 job/settled, 2556 real tool/result | 2557 step/end ...
session-11394456: ...2146 job/registered | [resume 11:44:34] 2147..2150 closers+end-seed | [stale writer 11:44:38 — 4.2 s later] 2147 job/settled, 2148 real tool/result, 2149 step/end, 2150 step/start | 2151 assistant/message ...
session-6966981f: ...1476 turn/end{completed} | [resume 11:57:49] 1477 session/end-seed | [stale writer 12:25:24] 1477 job/settled | 1478 agent/inbox/spliced ...

The third one has no interrupted turn and no synthetic closers: the constructor's session/end-seed at seq=log.length collided with the live writer's next event. So the collision only needs an open while a previous writer is still live.

Reader rejects with: "released v2 row 2557 has seq gap (expected 2557, got 2553)" etc.

Three suspects in this build: (1) the lease is acquired lazily at first durable write (persistContiguous :226, flush :135), not at write-open as the module doc :637 states, so AgentLoop.resume (dsh-agent-loop/lib/index.js:1986-2003) reads/appends/seeds in a lock-free window; (2) a released lease is never cleared from the handle (ensureLease :250 is ??=, release :724 leaves the field set) while drainBuffered (:211-214, enqueueChain :252) bypasses the closed-handle refusal, so a post-release drain writes lock-free; (3) persistContiguous (:227) validates the batch only against state.cursor and never against the artifact's actual length, so a stale handle appends duplicate seqs silently — readCurrent (:105) guards the opposite direction.

Offline stitch (drop the resuming writer's closure/seed records, keep the live writer's, seq untouched) restores all three losslessly; they load via Session.fromRestore and open in the GUI.

[Hcc32]

补一条 rc.2 上的反例，可能推翻"rc.2 已由 SessionWriteLease 结构性修好"这个结论。

我这边（Windows，0.2.1-rc.2 常驻进程，08:05 才升级到 rc.5）有 3 个坏档，全部是"中断 → 恢复"，但没有第二个写者：续跑写的是同一 turn/step 的全新一段运行（assistant/message → tool/call → tool/result → step/end → step/start），不是老写者回填一条真结果。

盘上指纹（session/end-seed 由构造器补，落盘比闭包晚 6～9 秒，排除了同批写入）：

seq 1503  step/start   turn=29 step=20
seq 1504  step/end     turn=29 step=20        # 合成闭包
seq 1505  turn/end     {interrupted}
seq 1506  session/end-seed {}
seq 1504  assistant/message  turn=29 step=20  # 续跑退回 1504 = 闭包前的长度

即续跑会话的 seq = log.length 基准里不包含它自己刚写下的闭包。同进程、跨 16.5 秒、三种长度视图，互斥锁盖不住这一类。

若 SessionWriteLease 在 rc.2 确实生效，那这条路径绕过的是恢复流程内部的 seq 基准（agent-loop resume 里 storedCount = persisted.length + closers.length 与 end-seed 之后真实长度的错位），建议在 append 落盘前断言 event.seq === handle 当前长度。

完整现场与修复判据见 #11707。