Borrowed designs: the session log is a mini-WAL — five textbook answers, and the closed channel in between

[MedicineKing]

Borrowed designs: the session log is a mini-WAL — four corruption shapes, one write-path characteristic, and the closed channel in between

(#5169) This design discussion is filed as the reader-side follow-up to the lease-coordination point argszero confirmed in #4942 (comment 18191847), and carries over the reader-side result table of this thread's measurement section. The prototype table below was re-run against the published 0.1.2-alpha.2 read path (@deepseek-ai/dsh-session 0.1.2-alpha.2, released 2026-08-30) — the rows are unchanged from rc.2, and that invariance is itself one of the points below. No new benchmark measurements — the point is that the four #4945 corruption shapes each correspond to a decades-old, named, standard mechanism, and that composing them yields a small protocol. Before that: one fact I found while checking the choice of carrier channel, because it is a release-notes inconsistency worth knowing regardless.

The observation

session.jsonl.zstd is already written like a transaction log: one checksummed zstd frame per append batch, a crash window between the final write and its durability point. What it is missing is the lifecycle protocol every real transaction log has — the part that defines "what is committed, and who wrote it". Today the read path answers those questions with symptom-level heuristics:

• torn tail → "final record without a trailing newline is dropped"
• stale cursor → "skip seqs smaller than the running index"
• recycled tail → "skip stale closers, then fold the end-seed recovery pair"

Each heuristic is a symptom-level answer to a question that has a definition-level answer elsewhere:

corruption shape (#4945)	textbook mechanism	current heuristic
partial tail after crash	WAL commit record (PostgreSQL XLOG commit, ext4 journal commit tag)	"last line has no newline"
two-writer interleave	fencing token / lease epoch (Raft term, Chubby fencing)	seq-monotonicity tolerance
recycled tail / stale closers	Kafka high watermark + epoch monotonicity	skip seq < index + end-seed fold
repair-tool splice (offline only)	content addressing (git object store)	seq-monotonic probe

The fifth mapped item — fsync per batch → PostgreSQL group commit (handle.sync() per append) — is a write-path characteristic, not a corruption shape; it composes with frame merging under #4949 and is returned to there.

The closed channel (release note vs code — rev. after 0.1.2-alpha.2)

The obvious carrier for protocol metadata is a new JSON record type, e.g. {"type":"log-commit","epoch":3,"endSeq":187}. Which is where I found a discrepancy — and, four days later, its reversal:

• rc.2 published build (@deepseek-ai/dsh-session 0.1.1-rc.2, the official read path maintained in the Regression gate for the session layer — a corruption corpus that would have caught this entire bug family #4945 gate): the vocabulary contract note says unknown types are refused "unless the event carries the envelope's ignorable marker … such a log was likely written by a newer harness". The ignorable flag is still validated (vendored lib/index.js — the seed check accepts ignorable:true).
• 2026-08-25 note (fail-closed session event vocabulary): the ignorable field was removed. "每个会话事件类型都是读取必需项" (every session event type is a read-required item) — PersistenceCoordinator compares every event type against the generated KNOWN_SESSION_EVENT_TYPES and rejects any unknown type with SessionFormatUnsupportedError, listing the index and pointing at "likely written by a newer writer". The note explicitly records that the skip-declaration was dropped because it had no production writer and added surface to every envelope.
• 2026-08-30 (0.1.2-alpha.2): that removal was reverted. The revert of PR 自定义调整会话压缩阈值 #3087 (remove-ignorable-session-events) plus new note retain-ignorable-external-session-events — ignorable is back end-to-end: packages/core/session/src/types.ts (ignorable?: true), known-event-types.ts (docstring now points at the 08-30 note), index.ts:224 (case 'ignorable'), and the SQLite path (one ignorable column per event row, so the field survives both the JSONL and the SQLite encodings).

So the "two truths" conflict was a five-day window (08-25 → 08-30), not a stable fork: rc.2 semantics and the main/a2 line are aligned again — the escape-hatch channel exists. What is still missing is not permission to add a record type; it is the semantics of commitment. ignorable: true buys a reader's tolerance for an unknown type; it does not define "last intact commit" or "who wrote this batch".

On the 08-25 deletion rationale — the two objections were no production writer and added surface to every envelope — both are answered by the shape of this proposal: a log-commit line is one record per batch, not per event (surface cost O(batches), not O(events)), and its production writer is the coordinator itself, i.e. this proposal supplies the writer the note said was missing. The flag is the carrier, not the protocol; the protocol below is the net addition.

Two channels

	P1: in-frame record line	P2: transaction sidecar
metadata	log-commit line at end of each batch	session.jsonl.zstd.meta manifest, atomic rename
compat with rc.2	subject to the unknown-type dispatch branch (fn.3 — it stops at the record today)	invisible — format untouched
compat with main	the coordinator may add an ignorable:true event type or a named non-event record category (both are listable entities on the guard's known-type check, like the session header record)	invisible — untouched
atomicity	frame-level (commit line is the last thing written)	rename-level (manifest last, or neither)
crash window	torn batch loses the commit line only	manifest lags → the reader cross-checks the manifest against the file and takes the more conservative claim — never stricter than a frame scan
fencing enforcement	reader-visible, strongest	writer-side (lease holder may update), reader verifies monotonic epoch
recommendation	the "true protocol" — ships together with the reader change	the "zero-dependency floor" — landable on an old codebase unfelt

Either way the protocol rules are the same:

The protocol (shared by both channels)

Read-side rules (4 rules: 3 replace the heuristics, 1 conserves today's behavior)

1. HW = last intact commit. In P1 the commit line's frame is the boundary; in P2 the manifest's claim is checked against the file (frame count/checksum) and the more conservative of the two wins. Commit records degrade like any other frame: HW is the last checksum-verified commit frame — if the newest commit frame is itself torn, HW steps back to the previous intact one, and to today's heuristic fallback when no commit frame exists at all. Everything after HW — newline or not — is uncommitted. The synthetic partial-tail-after-crash shape now yields the identical outcome as a corollary of HW, not of a string test.
2. Epoch monotonicity (fencing). Track maxEpoch; metadata with a lower epoch marks its batch as stale-writer residue and is skipped. This replaces both the seq < index skip and the stale-closer case — including the case the seq-based heuristic is weakest at, where the stale seqs coincide with the resumer's new tail. Epoch decides by who wrote it, not what seq it carries.
3. Epoch jump = switch. Higher epoch switches to the new writer and continues from its endSeq. This is the "recycled tail → ACCEPT" source: stale closers fenced, new tail resumes at seq 10 under the new epoch, contiguity follows without the end-seed fold.
4. Same epoch, conflicting seqs → fail closed. A writer that claims the current epoch but resumes with seqs colliding with the committed region keeps today's REJECT — the protocol never trades a live write for tolerance. (The "violating" variant in the table below is exactly this rule: same epoch, seq 10 expected 13.) Rule 4 is a conservation clause, not a new capability: it requires no new reader behavior, so the prototype below implements rules 1-3 (+fallback) and rule 4 is simply today's behavior stated explicitly and carried through.

Compatibility rule (stability guarantee): if neither channel is present in a log, a protocol-aware reader MUST use today's heuristics. Old artifacts are never judged more strictly.

Expected rows (per shape, under the protocol)

shape	with the protocol
partial-tail	ACCEPT (HW definition — same rows as today, now a corollary)
two-writer	ACCEPT — stale-epoch batch fenced; the current writer stays contiguous
recycled-tail	ACCEPT (rule 3, epoch-switch truncation semantics — no end-seed fold)
recycled-tail, violating (same epoch)	REJECT (rule 4 — fail-closed, never stricter)

The two-writer row is a deliberate softening of the #4942 expectation table, and it is explicitly on top of, not instead of the write-side lease fix: write-side prevents the interleave from occurring; this reader rule turns a leftover interleave into a no-op. The rc.2/alpha.2 ground truth for these rows is the measurement table below.

Measurement (prototype): the protocol reader vs the unpatched referee

I built a ~260-line zero-install prototype (same tooling style as the repair repo): a writer that emits the P1 batch layout, and a reader implementing the rules (plus the fallback). The four shapes from #4945 were generated as P1 variants (synthetic, content-free, deterministic) and run through both paths — the referee (verify-frame-format.mjs, unpatched official rc.2 read path, frame-level) and the P1 reader:

shape	unpatched rc.2 referee (frame-level)	P1 reader (rules 1-3 + fallback)
partial-tail (no commit frames written)	ACCEPT, 5 events contiguous	ACCEPT, 5 events (fallback — today's heuristics untouched)
two-writer interleave (A epoch 1, 0-9; stale B epoch 0, 4-8)	REJECT — seq 0…9 then seq undefined at the log-commit line	ACCEPT, 10 events (B's batch fenced: fenced=5 rows, A stays contiguous)
recycled tail, disciplined (prefix e0 0-9, closers e0 10-12, resumer e1 10-19)	REJECT — same referee stop	ACCEPT, 20 events (rule 3 truncated 3 prior-epoch events at seq>=10 — the 3 stale closers)
recycled tail, violating (resumer keeps epoch 0)	REJECT	REJECT — seq 10 expected 13 (rule 4; same failure as today)

Re-run against 0.1.2-alpha.2 (published npm build, 2026-08-30, via Session.fromRestore + official decodeStorageRecord — event-level):

shape	0.1.2-alpha.2 published read path
partial-tail	ACCEPT — restore OK (6 log events, liveSeq 5)
two-writer interleave	REJECT — seq 4 expected 10 (event-level stop; the rc.2 frame-level stop differs only by observation layer: seq undefined at the record line)
recycled tail, disciplined	REJECT — seq 10 expected 13
torn final frame (checksum-truncated)	ACCEPT, 0 recovered events at the Session layer (the torn final frame is dropped wholesale — that sample has no other intact frame to yield events; the persistence layer's torn-prefix recovery path itself is unchanged from alpha.1)
torn middle frame (magic damage)	REJECT — scanner throws at the damaged frame; no recovery path reached

The two runs disagree in the failed row's exit text but not in its semantics: the rc.2 column is the frame-level scanner contract (stops when a record line has no category), the alpha.2 column is the event-level fromRestore (reports the first conflicting seq) — both are REJECT for the same shapes, and both address the same question ("would the official reader accept this artifact?"). Where the tables report different counts for the same sample (5 events contiguous vs 6 log events / liveSeq 5), the accounting basis is the observation layer again: scanner-contiguous events vs Session.fromRestore loaded totals (header-derived state counted).

The negative datum: the published read path has not moved between rc.2 and alpha.2 — two consecutive releases, same REJECT rows. The alpha.2 release changed the vocabulary escape hatch (ignorable back), not the lifecycle semantics. The write-side lease gap (#4942 fourth point) and the reader rules here are still the live open items, and the #4945 gate remains unfilled for exactly these rows.

Three properties the table demonstrates, not asserts:

1. Fallback identity — with no commit frames (the crash window) the protocol decodes exactly as today; the protocol is never stricter than the current reader, it only relaxes where a commit frame exists.
2. Fencing is total — the two-writer shape flips REJECT → ACCEPT purely by epoch < maxEpoch; no sequence inspection was needed.
3. Stale closers are truncated, not folded — the recycled-tail shape is accepted without the end-seed fold via rule 3's later writer wins (epoch-switch truncation: on a switch, prior-epoch events at seq >= the new batch's first seq are dropped).

One referee-level artifact worth stating: the unpatched path stops at the log-commit line itself (seq undefined) — no released version has a category for it (the vendored reader evidently does not extend its ignorable tolerance to a record with no type category — the same dispatch-branch uncertainty as fn.3). That is the empirical form of the "closed channel" section (now: was closed for five days, reopened with ignorable): P1 artifacts are unreadable by today's reader by construction, so P1 ships together with the reader change, and only P2 can land on an old codebase unfelt.

The prototype script (prototype-wal.mjs) and the four artifacts are in my local tooling — I will attach them to the gate with the PR, alongside the extended verify contract.

Where this sits (and why it is not a duplicate)

• Proposal: persist the append watermark + per-session writer lock + lease before closers (closes the whole seq-corruption family) #4942 (comment 18191847) — argszero confirmed the lease-coordination gap as the fourth point of the write-side agreement. This post is the reader-side half of that point plus the commit/HW half: the epoch must be enforceable by read, or the lease is a gentleman's agreement.
• DSH: Why Opening a Large Cold History Session Can Stall the Whole Web Server — A Systematic Root-Cause Analysis and Optimization Roadmap #4416 roadmap — the A-stage sidecar work is where P2 folds in; the B-stage merge (frame count ↓, from Framing audit: per-frame costs dominate session cold-open — measured 15× decompress / 2.7× storage from frame merging (B-stage evidence for #4416) #4949's measured dominance) is where P1's fewer, larger batches live.
• Framing audit: per-frame costs dominate session cold-open — measured 15× decompress / 2.7× storage from frame merging (B-stage evidence for #4416) #4949 — group commit composes with frame merging: merging reduces frame count, group commit reduces fsync count; both target the same dominant write-path term.

Out of scope

• seq delta-encoding (saves bits; breaks self-describing frames and random access)
• full 2PC
• a new flag on every event envelope — no longer out of scope as a carrier: the 08-30 revert restored ignorable for external plugins, and the P1 log-commit line ships on top of it (one per batch, not per event). The flag is the carrier; the protocol rules (HW / fencing / epoch-switch / same-epoch fail-closed) are the real addition.
• byte-level frame-layout changes (zero migration holds for P2; P1 is a record line, not a layout change)

Cost

• P2: a small manifest writer (+atomic rename) and a ~20-line reader state machine; nothing about the existing file format changes.
• P1: same reader machine + either an ignorable:true event type or a named non-event record category (~10 lines, both listable on the guard's known-type check) + write-side epoch counter.
• Gate: the Regression gate for the session layer — a corruption corpus that would have caught this entire bug family #4945 pipeline already covers the four shapes (repair-splice as the offline probe). I'll extend the verify contract so both channels (and the fallback) are judged on the same artifacts; the before/after table becomes the review attachment. The repair-repo tooling already builds zero-install artifacts — a P2 sample and a P1 sample are each one generator case.

Footnotes

1. rc.2 vocabulary note — vendored 0.1.1-rc.2 lib/index.js:1064-1072; seed validation still accepts ignorable:true (lib/index.js:1229).
2. Reversal trail — 08-25 note .agents/notes/implemented/simplification/2026-08-25-fail-closed-session-event-vocabulary.md exists in the alpha.1 tree; absent in alpha.2, replaced by .agents/notes/implemented/architecture/2026-08-30-retain-ignorable-external-session-events.md (code: core/session types.ts/known-event-types.ts/index.ts:224,237). The revert commit (revert of PR 自定义调整会话压缩阈值 #3087, remove-ignorable-session-events) is merged into the alpha.2 release branch.
3. The rc.2 scanner dispatch branch for unknown types I read but will not claim line-level certainty on (that is why the P1 compat row in the channel table is qualified); the Regression gate for the session layer — a corruption corpus that would have caught this entire bug family #4945 gate is the executable check (both channels add one artifact each and the gate judges them by the official read path).
4. All existing artifacts remain synthetic and content-free per the Regression gate for the session layer — a corruption corpus that would have caught this entire bug family #4945 PII policy.
5. Alpha.2 re-run was executed against the published npm build (@deepseek-ai/dsh-session 0.1.2-alpha.2) via Session.fromRestore + the official decodeStorageRecord; the scanner-identity claim is from a line-level diff of zstd.ts:48-105 between the alpha.1 and alpha.2 tags.