Session log corruption when two harness processes share one DSH_HOME (stale live session appends after another process resumes)

[aka-danielZhang]

Summary

Two harness processes sharing the same sessions root (e.g. the default ~/.dsh) can silently interleave appends to one session's JSONL log. Every single-writer guard is per-process, and the JSONL append path validates only the coordinator's in-memory cursor, so the interleave is never detected at write time. The next cold load then refuses the log with corrupt session log: seq gap in committed region …, the web UI shows history unavailable … (internal), and the session becomes permanently unloadable — restarting does not help.

What happened (real incident)

A 44,802-event session log ended up with exactly one anomaly (lines abbreviated):

{"type":"turn/end","seq":6574,...}              // turn 1 completes in process A
{"type":"session/end-seed","seq":6575,...}      // process B resumes the session; the constructor's marker is persisted
{"type":"agent/inbox/spliced","seq":6575,...}   // 13 min later, process A's still-live session appends — DUPLICATE seq
{"type":"turn/start","seq":6576,...}            // A keeps running turns 2–7, locally consistent through seq 44800

Every cold load now fails with corrupt session log: seq gap in committed region at line 541 (expected 6576, got 6575).

Root cause

1. Session.append assigns seq from the in-memory log length: packages/core/session/src/index.ts#L629.
2. When process B resumes the session, the Session constructor appends session/end-seed at the seed boundary (index.ts#L545), and attachPrepared persists it as the unpublished suffix (coordinator.ts#L1184-L1207). The durable log now ends at seq 6575.
3. Process A's Session object is still alive (its browser tab stayed open) with log length 6575 — it never saw the marker. Its next append reuses seq 6575.
4. A's coordinator validates the batch only against its in-memory state.cursor (coordinator.ts#L698-L702) — 6575 === 6575 passes.
5. The JSONL backend opens the file with O_APPEND and writes at whatever the current EOF is (session-persistence-jsonl/src/index.ts#L651-L654). The duplicate lands right after B's marker. No cross-process check exists anywhere on the append path.

All single-liveness enforcement is in-process: prepare() refuses a second live session only via the local ctx.sessions store (coordinator.ts#L724), and the onCreated collision logic uses the coordinator's per-process maps. Nothing coordinates two OS processes sharing one root.

Why I believe the scenario is legitimate

• Multiple harness processes on one DSH_HOME is a realistic setup: the documented commands have users run dsh web alongside pnpm dsh --profile headless "task" or the ACP server, and nothing in the docs declares a sessions root single-process-exclusive.
• The codebase already anticipates concurrent processes elsewhere: materializePosix publishes via link() precisely so that "two processes materializing the same id concurrently cannot clobber each other" (index.ts#L544-L546), and read paths retry on revision changes because "continuous external writers may delay completion".
• The SQLite backend already fails loud in this situation: PRIMARY KEY (session_id, seq) (schema.ts#L145) turns a stale append into an immediate UNIQUE violation. Only the JSONL backend corrupts silently.
• The failure mode is disproportionate: silent at write time, discovered much later as a "corrupt session log" error that looks like disk damage, and the loader (by design) offers no repair.

Minimal reproduction

Two independent Context instances over one JSONL root faithfully emulate two processes (each has its own SessionStore/coordinator; the filesystem is the only shared channel — exactly as with two OS processes):

// packages/session/session-persistence-jsonl/tests (vitest)
it('two coordinators sharing one root corrupt the committed region', async () => {
  const ctxA = await boot(root) // SessionStore + JsonlSessionPersistence
  const a = ctxA.sessions.create(ID, { meta: { cwd: '/w' } })
  a.append('turn/start', { turn: 1 })
  a.append('turn/end', { turn: 1, reason: { kind: 'completed' } })
  await ctxA.sessions.flush(a) // disk: seq 0,1 — session stays live in "process" A

  const ctxB = await boot(root)
  const prep = await ctxB.sessionPersistence.prepare(ID) // constructor appends end-seed (seq 2)
  ctxB.effect(function* () {
    yield ctxB.sessions.enter(prep.session)
    ctxB.sessions.announce(prep.session) // attachPrepared persists the marker
  }, 'resume')
  await ctxB.sessions.flush(prep.session) // disk: seq 0,1,end-seed(2)

  a.append('turn/start', { turn: 2 }) // stale A assigns seq 2 again
  a.append('turn/end', { turn: 2, reason: { kind: 'completed' } })
  await ctxA.sessions.flush(a) // in-memory cursor check passes; O_APPEND interleaves

  const ctxC = await boot(root)
  await expect(ctxC.sessionPersistence.load(ID)).rejects.toThrow(/seq gap in committed region/)
})

Resulting file:

{"type":"turn/start","seq":0,...}
{"type":"turn/end","seq":1,...}
{"type":"session/end-seed","seq":2,...}
{"type":"turn/start","seq":2,...}   // duplicate seq, now inside the committed region
{"type":"turn/end","seq":3,...}

Candidate fix directions

1. Optimistic revision check on the append path. The coordinator already has source-qualified revisions (readStoredRevision). Remember the revision at which each in-memory cursor was established; before each appendBatch, re-read and compare; on mismatch, reject loudly ("another process is appending to this session log"), and refresh the remembered revision after each successful write. O(1) stat per batch, backend-agnostic, and turns silent corruption into an immediate, actionable error. A check-then-write TOCTOU window of one batch remains.
2. Cross-process exclusion (stronger, more invasive): a lockfile/lease per session directory so a second process cannot resume a session another process still owns. Brings stale-lock reclamation and Windows semantics questions.
3. Even if shared roots are declared unsupported, (1) plus an explicit documentation note would replace a late, cryptic corruption error with an immediate one that names the actual cause.

I maintain a fork and have implemented direction (1) with regression tests; I will link the branch in a comment below. Happy to open a PR if you are interested.

---

中文摘要：两个共享同一 ~/.dsh 的 harness 进程可以向同一会话的 JSONL 日志交错写入：进程 B 恢复会话时把 session/end-seed 落盘，进程 A 里仍然存活的旧 Session 对象对此无感知，继续用 seq = 内存日志长度 分配序号；协调器只校验内存游标，appendLines 用 O_APPEND 写到文件当前末尾，于是产生重复 seq。写入时完全无感，之后的冷加载按设计拒绝「已提交区域」的缺口，会话永久无法打开，重启无效。SQLite 后端靠主键约束会在写入时直接报错，只有 JSONL 后端静默损坏。我在 fork 里实现了「写前比对 durable revision」的乐观检查作为修复，链接见下方评论。

[aka-danielZhang]

Reference implementation on our fork: branch fix/session-persistence-concurrent-writer, single commit 647c94b (direction 1 from the post above).

What it does:

• PersistenceCoordinator records the source-qualified durable revision at which each per-id cursor was established (in commitPrepared and live-prefix adoption) and refreshes it from readStoredRevision after every successful appendBatch.
• appendCore re-reads the revision before writing and rejects on mismatch: session "…" log was advanced by another writer … refusing to append — another harness process is sharing this sessions root.
• A rejected appendBatch must leave the stored log unchanged (the hook contract now states this rollback obligation), and the failure path re-establishes the baseline so a transient I/O failure stays retryable — pinned by the pre-existing rollback-retry tests, which pass unchanged.

Verification:

• The reproduction above became a shared coordinator-contract scenario (two contexts over one storage scope ≡ two processes) that runs against the memory, JSONL (raw and zstd), and SQLite backends: the stale flush rejects, and the stored log still loads cleanly.
• The SQLite mid-transaction rollback test now calls the appendBatch hook directly, because the coordinator guard legitimately refuses its former service-level scenario first.
• Persistence suites: 443 tests green; strict typecheck (exactOptionalPropertyTypes) green; no public interface changes.

Known residual: the check-then-write window of one batch, documented in both persistence READMEs. Full exclusion would need a cross-process lock (direction 2) — happy to explore that next if you want it.

---

中文：fork 参考实现见上面的分支与 commit 链接。协调器为每个会话记录「游标建立时的持久修订值」，每次 append 前复核、写后刷新；发现日志被其他进程推进就以指明原因的错误拒绝写入，瞬时 I/O 失败回滚后仍可正常重试。四个后端的协调器合约套件全绿，公开接口零改动。残余的一个批次级 check-then-write 窗口已写进两个持久化 README。

[aka-danielZhang]

Follow-up on contributing this back: I attempted to open a cross-fork PR, but CreatePullRequest is rejected for my account on this repository (both GraphQL and REST), so external PR creation appears to be restricted to members. If you would like the change as a PR, either relax that restriction and I will open it, or pull the ready branch directly:

• Branch based on current master (99f6f02), one commit: fix/session-persistence-concurrent-writer-upstream — commit 4bf99f8
• git fetch https://github.com/aka-danielZhang/deepseek-harness.git fix/session-persistence-concurrent-writer-upstream

中文：跨仓库 PR 被仓库权限拒绝（非成员无法创建），如需合入可直接抓取上面的分支（基于当前 master 的单提交），或放开限制后我来提。

[aka-danielZhang]

Two more real-world data points from my logs — same root cause, one new variant, and one reader-side robustness gap worth knowing about.

Variant: crash-repair × resume interleaving with a live foreign writer (offset −3). In two more sessions the damage was:

step/end   seq=52672  turn=8   time=...953637   ← synthesized by process B's cold load
turn/end   seq=52673  reason=interrupted  (same ms — synthetic closers)
session/end-seed  seq=52674  time=...956123     ← B's resume marker
text-chunks  seq0=52672  turn=8 step=14  time=...963190  ← process A's still-live turn keeps streaming
... A then runs the turn to a genuine completion (and, in one log, NINE more full turns)

Process B cold-loaded a session whose turn was open, synthesized the interrupted closers, and resumed it — all while process A's live session was still mid-turn. A's next batches were 3 seqs behind (closers + marker), exactly the class the revision guard catches. Note B did nothing wrong locally: load() cannot see A's live Session across processes.

Reader-side observation: the zstd path misclassifies an uncommitted seq gap as physical corruption. When the interleaved tail contains no later turn/end, the scanner tolerates it as an uncommitted tail and freezes committedBytes — but readZstdPrefix then hits checkpoint().committedBytes !== inputBytes and throws corrupt Zstandard session log: complete frame contains a torn JSONL record. The plaintext path treats the same logical situation as a truncatable torn tail and loads the committed prefix. So the zstd reader is stricter than the format's own repair policy, and the error message points at the wrong cause. Consider routing that checkpoint mismatch through the same uncommitted-tail truncation (it only fires after the scanner has already vetted committed-region integrity), or at least a distinct message.

Both logs repaired locally: one by dropping the orphan tail (the 0457 case above), two by deleting the three disproved synthetic/marker lines — the live writer's continuation was byte-contiguous with the pre-closer log, so no renumbering was needed. The revision-guard branch prevents all of these at write time.

[LLM-X-Factorer]

补一组发布版 rc.7 的独立、无模型 A/B 复现，结果与本帖一致。

环境：macOS arm64、Node 25.8.2、@deepseek-ai/dsh@0.1.0-rc.7，所有 Harness home 和 session root 都是一次性目录。

共享 root

两个 live persistence owner 打开同一 root 下的同一 session。第二个 owner 恢复并推进日志后，第一个 stale owner 的后续 append 没有被拒绝。最终持久事件序列为：

[0, 1, 2, 3, 3, 4, 5]

后续 cold load 拒绝该日志，分类为 committed region 中的 seq gap。

隔离 root 对照

两个 owner 使用相同 session id，但分别写入两个独立 root。两边均可冷加载，事件序列都是：

[0, 1, 2]

因此 session id 相同不是触发条件；关键条件是多个 live writer 共享同一可写 session root。这个对照也支持当前操作规避：Web、headless、ACP 和自动测试分别使用独立 DSH_HOME，每个可写 root 同时只允许一个 owner。

[denial123789]

I turned the rc.7 evidence in this thread into a source-backed operator runbook: https://sandbaseai.github.io/deepseek-harness-handbook/deepseek-harness-single-writer.html

The operating boundary is intentionally narrower than “shared root always corrupts”: the dangerous condition is two independent live processes reaching the same Session artifact. Their coordinators, liveness maps, and cursors are process-local. The official persistence README also states that its revision freshness check does not add cross-process writer exclusion.

The guide therefore gives two explicit deployment choices:

• concurrent Web, headless, ACP, SDK, service, or test processes use isolated DSH_HOME values or isolated configured persistence roots;
• processes that must share history serialize ownership: stop admission, drain work, terminate gracefully, verify the previous PID exited, then cold-load from the next owner.

It also calls out two easy traps: different ports do not imply different storage, and a closed browser tab does not prove the Web Host stopped. After suspected overlap, the runbook stops all writers and preserves offline copies instead of presenting event renumbering or live-log edits as a general repair. All runtime claims link to the pinned rc.7 source. Thanks for the incident and independent A/B evidence.