Proposal: persist the append watermark + per-session writer lock + lease before closers (closes the whole seq-corruption family)

[MedicineKing]

A write-side proposal addressed to maintainers, moved out of the #1497 thread so it can be tracked as a topic. Every report in the session-corruption family — #1333, #1452, #1497, #1586, #2068, #2167, #3198, #4274, #4662 (canonical list, see the verification-side proposal) — is produced by one of three write-side gaps, and all three are fixable without touching the reader:

1. Persist the append watermark with the durable prefix. Recovery resumes from the in-memory log.length of an old Session instance instead of the committed cursor — the source of the crash-replay class. A watermark value committed inside the same batch as the events (or derived from the committed prefix length) removes the guess.
2. Cross-process writer exclusion per session. appendLines = open(path, 'a') + write + fsync with no cross-process coordination, so a second dsh process (web + desktop, or two web ports sharing DSH_HOME) duplicates seqs deterministically. A per-session lock (O_EXCL lockfile / advisory lock held for the append+fsync window; refuse writes if content changed since last read) closes it — the direction nokkies' branch in 会话日志并发写损坏修复：跨进程写锁 + 尾部 seq 校验（附完整修复分支） #4662 already demonstrates.
3. Recovery must not synthesize closers while the old executor may be alive. The interruptedTurnClosers path assumes it is the only writer; a lease/heartbeat check (or at minimum: do not create the closers until the in-flight turn is provably dead) removes the [bug] 会话历史加载失败：崩溃恢复与残留执行流并发写同一日志，seq 重叠（corrupt session log: seq gap in committed region） #1586 overlap class.

With those three, the reader never has to tolerate anything and no session ever needs file-level surgery.

I am happy to test a candidate branch against a real corruption corpus (multi-frame zstd sessions from this family, ~20 samples) and publish the verification numbers if it helps.

In the meantime the read-side tolerance patch and the zero-install CLI in the #1497 thread remain the safe stopgaps — nothing in this family has ever lost user data: the bytes stay, only the pointers disagree.

[MedicineKing]

Related prior art in this category, for the record: #718 catalogs several write-path losses in the same layer (rollback truncating others' bytes, coordinator loop under concurrent appends), and #1518 argues for a user-facing session doctor — this proposal is the complementary piece they both point at: fix the writer (three gaps), so neither the doctor nor the reader ever has to work around it.

[argszero]

Source-backed confirmation of all three write-side gaps against alpha.1 (HEAD cd5ef81481). Your three-part framing is exactly right, and each gap has a precise in-tree site:

Gap 1 — append watermark / committed cursor. packages/core/session/src/index.ts:627 still derives the next seq from the running instance's in-memory length: seq: this.log.length (also :537 this.firstLiveSeq = this.log.length, :564 return this.log.length). The recovery path doesn't consult a persisted committed cursor — coordinator.ts:984 feeds interruptedTurnClosers(storedEvents) the durable prefix and starts the synthetic close from last.seq + 1. Two writer paths → two cursors, exactly as you say.

Gap 2 — no cross-process writer exclusion. session-persistence-jsonl/src/index.ts:673 appendLines: open(path, 'a') + writeFile + handle.sync(), and the rollback (:689→:700) is a same-process truncate. There is no O_EXCL lockfile, advisory lock, or "content changed since last read" check anywhere in the backend's write path — a second process appending the same file deterministically duplicates seqs (the :684 append races the other writer's bytes, and :685 fsync does not coordinate them).

Gap 3 — recovery synthesizes closers assuming it's sole writer. coordinator.ts:984-985 runs interruptedTurnClosers during repair and publishes the closers via commitRepair, with no lease/heartbeat probe for a still-living old executor. That's the #1586 overlap class I analyzed at source (both a persisted-cursor recovery writer and an in-memory-cursor live writer produce the same seq; the decoder's committed-region check at session-persistence-jsonl/src/index.ts:388 then rejects the whole tail). This one is the least solvable by read-side tolerance alone, because the closers are legitimately "logical" events the reader cannot distinguish from real ones without a third fact.

One addition to your three, worth flagging as a fourth consideration: even with a lease, the timing of the lease acquisition matters for the crash-tail case. appendLines writes then fsyncs, and the rollback truncates back to before. A writer that crashes between writeFile and rollback (i.e., after partial bytes are on disk but before the cursor advances) leaves partial bytes at the tail — :666-668 already documents this ("leaving partial bytes would create duplicate sequence numbers"). A per-session lock held across the whole append+fsync window, plus a durable cursor written in the same batch as the events, closes that window too — but they must be co-designated with the lock, not added independently, or the lock can serialize writers while the durable cursor still lets a crashed writer's partial tail be misread.

I'd be glad to test a candidate branch against the ~20-sample corruption corpus you mention — that's the right way to validate a write-side fix, because jsdom/single-process tests cannot exercise the multi-process interleaving these three gaps depend on.

中文摘要：已在 alpha.1 源码确认三处写侧缺口——① index.ts:627 seq: this.log.length（内存游标，非持久化 watermark）+ coordinator.ts:984 恢复路径从 last.seq+1 合成；② session-persistence-jsonl/src/index.ts:673 open(path,'a')+writeFile+fsync，全 backend 无 O_EXCL/advice/变更检查；③ coordinator.ts:984 interruptedTurnClosers 恢复合成闭包前无 lease/存活探测（#1586 重叠类）。另提第四点：lease 与持久化 cursor 需协同设计（writeFile 后 rollback 前崩溃会留部分字节尾，:666-668 已文档化），否则锁能串行化写者但部分写仍可能被误读。愿用 20 样本损坏语料测候选分支——写侧修复的多进程交错正是单进程测试看不见的。

[MedicineKing]

@argszero — thank you, source anchors and all. It is exactly why we wrote the proposal the way we did: we would rather be held to the in-tree checks than to our own summary.

On your fourth point: fully agreed, and I'd go a step further — the co-designation is what makes the window actually closed. Lock-alone serializes writers but leaves the crash-tail interpretable as committed bytes (the durable cursor says the batch exists); cursor-alone reorders writers but leaves the partial tail to interleave. They are one feature with two sides, not two features.

Consistent with that, the corpus will carry an explicit partial-tail-after-crash catalog row (torn last record + later real events; a synthetic variant of exactly this shape will be the public row — the real artifacts stay local per the PII policy), so a candidate branch has to define — and its tests have to meet — its own expectation for it. That same row doubles as the regression case for the Gap-3 lease (closers-vs-live-writer overlap), which we agree is the least solvable read-side — and we do not claim it is.

Test pipeline status: 20+ real artifacts (PII-safe metadata) + injector for the enumerated shapes + official-path referee (Session.fromRestore / assertZstdHeaderFrame / scanner contract) + per-artifact table output; single command, zero install. When a candidate branch lands, give me the merge point and I will run it — and I'll also ship a two-process interleaving row (two writers, one session), since that is precisely the interleaving class single-process tests cannot see, as you said.

[MedicineKing]

Follow-up on v0.1.2-alpha.1: gap check against #4942, with a measured referee table

Ran the agreed corpus check against the newly-merged alpha.1 release point (cd5ef81, dsh@0.1.2-alpha.1) — 0.1.2-alpha.1 sources for the full read/write path, and the same 4 shapes from the #4945 catalog, judged by both the published 0.1.1-rc.2 read path and the 0.1.2-alpha.1 one. Outcome: two of the write-side gaps landed in alpha.1; the two-writer/lease gap did not.

Net: alpha.1 closes 2/4 — read-side torn recovery + materialize clobber-prevention; write-side append lock + lease/epoch remain open (per #4942 gaps 2-4).

What landed in alpha.1 (verified in the tree + measured)

1. Torn-write recovery is now a system, not a heuristic. readZstdPrefix → scanZstdFrames (which now returns tornStart instead of failing) → decompressZstdPrefix (flush-only decode → complete records) → commitRepair (truncate + re-append recovered + closers). This is the read-side model described in Bug: corrupt session log after unclean exit — replayed events cause seq gap in committed region, history unrecoverable #1497/Proposal: persist the append watermark + per-session writer lock + lease before closers (closes the whole seq-corruption family) #4942, implemented natively. Measured: a frame-level torn variant (last 4 bytes = checksum only, data intact) — rc.2 read path REJECTs (structural error), alpha.1 salvages 5/5 complete records from the flush-only decode. Cutting deeper into the payload returns 0 records (the boundary is exactly the checksum), which is what you'd expect from a flush-only recovery.
2. Materialization clobber-prevention — materialize publishes via link()+unlink() (EEXIST if the path exists) + rejectExistingLog + exclusive wx temp. Two processes materializing the same id can no longer clobber each other. Also readStableFile's revision-stable loop absorbs concurrent-append races.

What did not land

3. Same-session two-writer append: unchanged. appendLines is still bare open('a') → write → sync(). No lock, no lease, no epoch. The corpus row behaves identically in alpha.1: two-writer-interleave still REJECT at seq 4 expected 10 — same line, same failure, same referee output as rc.2 (table below).
4. Lease/epoch: not in the tree at all. Zero occurrences of any epoch/lease/HW concept in the persistence path. The write-side coordination point from Proposal: persist the append watermark + per-session writer lock + lease before closers (closes the whole seq-corruption family) #4942 remains entirely open.

One thing worth confirming about the recovery path itself. commitRepair is also a writer: it truncates to the torn marker, re-appends the recovered events and synthesizes the closers — without acquiring anything. If the original writer is still alive (a torn frame is not only caused by a crash; stale cursors and interrupted flows also produce one), recovery and the old writer are exactly the two-writer case above, and the synthesized closers are exactly what the recycled-tail shape is made of. Recovery probably runs on next open by a fresh process, and then this is a non-issue — but it inherits the same implicit single-writer assumption on the coordination side, so worth confirming before the lease design lands: should the lease-fix cover the recovery writer too, or is recovery strictly a new-process path?

Measured referee table (same artifacts, both readers)

shape	rc.2 (published)	alpha.1
partial tail (row-level torn)	ACCEPT, 5 events contiguous	ACCEPT, 5 events contiguous
frame-level torn (checksum cut)	REJECT — structural error	ACCEPT — frame dropped, 5 records salvaged by flush-only decode, session restores
two-writer interleave	REJECT — seq 4 expected 10	REJECT — same line, same failure
recycled tail (stale closers)	REJECT — seq 10 expected 13	REJECT — same line, same failure

Note on the first two rows: the alpha.1 row-level-torn row is the same outcome as rc.2 only because the synthetic shape is a row-level tear inside a structurally complete frame — the alpha.1 torn machinery only kicks in at the frame level, where it is strictly more tolerant. Neither reader has changed its judgment on interleave/recycled.

Gate status

The #4945 gate now has an alpha.1 referee variant (same zero-install style; imports the alpha.1 dsh-session from the release tree — 0.1.2-alpha.1). The frame-level-torn shape is now a corpus row (deterministic, synthetic, content-free, +1 generator case): it documents the one row where the two published readers disagree on the same artifact — which makes it the highest-value row in the new table. Happy to attach the full run output to the merge review when a lease-fix branch appears.

Still holding for the write-side fix branch per the earlier agreement — and when it lands, the gate will now report both readers, so the "read-side tolerance vs write-side prevention" split is measured, not argued.

[denial123789]

Expanded the handbook Session-corruption runbook with the write-side prevention model: persist an append watermark with the durable prefix, exclude concurrent writers through append+fsync, and require an executor lease before synthesizing recovery closers. The guide also calls for a multi-frame corpus and two-process shared-DSH_HOME test. https://github.com/sandbaseai/deepseek-harness-handbook/releases/tag/v0.5.393

[argszero]

On your question — "should the lease-fix cover the recovery writer too, or is recovery strictly a new-process path?" — the source says recovery is not strictly a new-process path, and it's worth pinning the exact sites because there are two distinct recovery entry points with different safety postures.

Entry point 1 — open/load recovery (the fresh-process path). session-persistence/src/coordinator.ts:1026 runs commitRepair from commitPrepared, which is driven by reading a stored prefix that has a torn marker. This is the path your #4945 corpus exercises and the one where "new process opens, sees torn tail, repairs it" holds. Here recovery-on-open is safe because there's no live session bound yet — I'd argue the lease requirement here is only that the recovery writer is not concurrently racing an already-open live session, which the states.get(id)?.owner !== undefined check at :1011-1012 partially gives you.

Entry point 2 — adoptLivePrefix (reload/HMR/reload-data), coordinator.ts:1392. This is the one that breaks the "strictly new-process" assumption. When a session is already live and the harness reloads the stored prefix (HMR of the dsh process, or session load on an open session), adoptLivePrefix calls this.backend.commitRepair(meta, tornMarker, []) at :1392 with empty closers — the comment explicitly says "Truncate-only repair (no closers): the open turn is NOT closed here." So recovery DOES run while a live session holds the same id, and it's not a fresh process: the live Session object is still the authority (:1395 binds owner: session). The docstring even warns the live Session "is still the authority" — which is precisely the two-writer overlap.

So the honest answer is: recovery is not strictly a new-process path once adoptLivePrefix is involved. The gap-3 overlap (synthesized closers vs a live writer) that we agreed is the least solvable read-side exists here too, but in a worse form: instead of closers racing a live writer, you have a truncate racing a live writer.

The most important asymmetry — JSONL has no staleness guard, SQLite does. Your commitRepair in the SQLite backend (session-persistence-sqlite/src/store.ts:214-232) is already defensive: it rejects with repair is stale: physical tail no longer starts at seq ${tornMarker} when the current tail no longer matches the recorded torn marker. But the JSONL backend's repair() (session-persistence-jsonl/src/index.ts:711) is a blind truncate(path, offset) with no equivalent check — it truncates to offset regardless of whether the file has grown since the marker was recorded. A lease that only covers the append writer doesn't protect against this: a recovery writer that truncates a file that a live writer has since appended to is exactly the destructive overlap the SQLite guard is designed to catch, and JSONL is missing it.

Concrete implication for the lease design. I'd frame it as: the lease must cover every writer on that log, and JSONL's repair() needs an analog of the SQLite staleness check — capture the file identity/size at the moment scanZstdFrames read the marker, and before truncating, verify the tail still begins at the recorded truncateTo offset (or carry an epoch). Otherwise a lease on appendLines serializes appends but the recovery/reload path (:1392) can still destructively truncate bytes a live writer just wrote.

One refinement I'd push back on, gently: the distinction isn't "new-process vs live" — both :1026 and :1392 can run against a log that a live writer owns. The real invariant to preserve is "repair must never shrink a log that has grown past what the repairer observed." That's what the SQLite staleness check enforces and what JSONL's repair() currently does not.

中文摘要：回答维护者的"lease 是否应覆盖恢复写者"问题——源码显示恢复并非严格限于新进程路径。两个触发点：① coordinator.ts:1026（open/load 恢复，commitPrepared→commitRepair，此处无 live session 绑定，states.get(id)?.owner !== undefined 检查部分给出保护）；② adoptLivePrefix coordinator.ts:1392（reload/HMR，session 已 live 时 commitRepair(meta, tornMarker, []) 空 closers 截断修复，注释明说"open turn 不在此关闭"、live Session 仍是权威）——这打破了"新进程"假设。关键不对称：SQLite 的 commitRepair（store.ts:214-232）已有 stale 防护（repair is stale: physical tail no longer starts at seq N），但 JSONL 的 repair()（index.ts:711）是无条件盲 truncate(path, offset)，无等价检查。因此 lease 必须覆盖每个写者，且 JSONL 的 repair() 需要 SQLite 的 stale 检查（截断前验证 tail 仍起始于记录的 offset，或携带 epoch）。真正的修复不变量不是"新进程 vs live"，而是"repair 永不将已增长超过观察点的日志缩小"。

[MedicineKing]

reply to argszero's source-level response (comment 18193453 follow-up thread, discussion #4942)

@argszero — this is exactly the kind of answer we wrote the question for. Adopting your framing wholesale:

The invariant is not "new-process vs live"; it is "repair must never shrink a log past what the repairer observed." Both :1026 and :1392 can run against a log a live writer owns, and adoptLivePrefix (:1392, truncate-only, empty closers) is the worse form of it: a truncate racing a live writer, not closers racing one. Your distinction is sharper than ours, and it changes the lease requirement statement — I'll use it in any further write-ups: the lease must cover every writer on the log, including both repair entry points.

The JSONL/SQLite asymmetry is the stand-out for us. SQLite's store.ts:214-232 refuses a stale repair; JSONL's repair() (index.ts:711) truncates blindly. Our repair tool has the same shape of hazard to avoid on its own side (it only rewrites whole frames and never truncates, so it does not share the bug — but it will adopt the "verify before touching" rule for consistency). For the gate: the read-side manifestations of both entry points fall inside the torn family our corpus already covers (partial-tail / frame-level torn / torn-middle — the published four rows plus the new torn-middle row, all judged by both rc.2 and alpha.1). What the blind truncate adds is a write-side stale-check test — best expressed as a unit test on the future fixed repair() than as a new corpus shape, since the corpus judges artifacts, not timings. If a candidate branch lands with the stale check, we'll run the full set (20+ rows) and the reload-race row can be a test-side attachment.

Two things we will do:

1. Update the corpus table note to record this answer (two entry points, the invariant, the asymmetry) so future readers get the source-pinned story.
2. When the branch lands: full-corpus before/after, plus a rerun of the torn-middle row — the shape where alpha.1 currently hard-fails (no frame magic at 356, from our probe run), which the stale-guarded repair should make unreachable, not merely rarer.

We are happy to hold the answer until your write-side branch is ready for the 20-sample run per the earlier agreement.