[Bug] A session log that lost its final frames reads back as a complete session

[Kizaemon]

Versions: DSH 0.1.7-rc.2, Node v22.23.2, Windows Server 2022. Package: @deepseek-ai/dsh-session-persistence-jsonl@0.1.7-rc.2 (lib/index.js, SHA256 0845707017ACC2B4A8A75EAB2244FA3FD88587B8094AB32014321DCE7CA1E31B).

Summary. A session log is a concatenation of independent Zstandard frames. If a log loses whole trailing frames, the reader returns a shorter but perfectly valid session: every frame checksum validates, the surviving records parse, and their seq numbers are contiguous from 0. Nothing surfaces an error, so a truncated transcript is indistinguishable from a session that ended earlier. Damage inside a frame is caught; loss of complete frames at the tail is not.

Why it happens. Four properties combine:

1. scanZstdFrames (index.js:1300) walks frames by index with no declared total, so the frame list simply ends earlier. A structurally valid prefix is not distinguishable from a complete file.
2. No frame declares its plaintext size. compressZstdFrame passes only CHECKSUM_OPTIONS (index.js:1291, 1369); the observed frame header is descriptor 0x04 → contentSizeFlag=0, singleSegment=0, so the reader's header parse (index.js:1318-1324) has no per-frame length to verify against.
3. The header carries no completeness witness. Observed keys are type, version, id, createdAt, cwd, parentSession, isSeeded, origin, delegationDepth, agentPreset — no event count, byte offset, or total. (inheritedEventCount exists only for seeded/forked logs and describes the inherited prefix, not the file.)
4. Sequence continuity cannot detect it. Tail loss removes events from the end, so what remains is seq 0..N contiguous. The contiguity checks (assertContiguous, dsh-session-persistence/lib/index.js:229) validate appends against the cursor derived from the same shorter read, so appends resume seamlessly after the gap.

In-process case is already handled — the gap is the restart case. JsonlSessionHandle.observedLength (index.js:36, 102-105) already refuses a shrunken log:

if (source.events.length < this.observedLength) throw new Error(`session "${this.id}": stored log
  shrank below a previously observed prefix (${source.events.length} < ${this.observedLength})`);

That covers truncation while the same process holds the handle. Because observedLength is a per-handle field starting at 0, a new process accepts the short log silently — and readers outside DSH never had the guard at all.

Reproduction. No DSH code or fixture is needed. Against any session*.jsonl.zstd under the session store, cut the file at a frame boundary and compare. Measured on a real 221-frame v4 log (435,248 bytes):

cut	frames	records	last seq	error surfaced
full file	221	456	454	—
drop last frame	220	453	451	none
drop last 20 frames	201	421	419	none
cut inside final frame	220 + torn	453	451	none (prefix recovered by readZstdPrefix)

Drop-last-frame, showing that nothing complains:

import { readFileSync, writeFileSync } from "node:fs";
import { zstdDecompressSync } from "node:zlib";

const MAGIC = 0xfd2fb528;
function frames(b) {                       // walk frames structurally, as scanZstdFrames does
  const out = []; let p = 0;
  while (p < b.length) {
    const s = p;
    if (b.length - p < 4 || b.readUInt32LE(p) !== MAGIC) break;
    p += 4;
    const fd = b.readUInt8(p); p += 1;
    const fcs = fd >>> 6, single = (fd >> 5) & 1, ck = (fd >> 2) & 1, df = fd & 3;
    p += (single ? 0 : 1) + (df === 3 ? 4 : df) + (fcs === 0 ? (single ? 1 : 0) : 1 << fcs);
    for (;;) {
      const bh = b.readUIntLE(p, 3); p += 3;
      const last = bh & 1, type = (bh >>> 1) & 3, size = bh >>> 3;
      p += type === 1 ? 1 : size;
      if (last) break;
    }
    if (ck) p += 4;
    out.push({ start: s, end: p });
  }
  return out;
}
function read(b) {                          // DSH's contract: decode each frame, then split on raw LF
  let text = "";
  for (const f of frames(b)) text += zstdDecompressSync(b.subarray(f.start, f.end)).toString("utf8");
  const recs = text.split("\n").filter((l) => l.length > 0).map((l) => JSON.parse(l));
  const seqs = recs.filter((r) => typeof r.seq === "number").map((r) => r.seq);
  return { records: recs.length, firstSeq: seqs[0], lastSeq: seqs.at(-1),
           contiguous: seqs.every((s, i) => i === 0 || s === seqs[i - 1] + 1) };
}

const path = process.argv[2];
const buf = readFileSync(path);
const f = frames(buf);
console.log("full           :", read(buf));
writeFileSync("truncated.zstd", buf.subarray(0, f[f.length - 2].end));   // drop final frame
console.log("minus last frame:", read(readFileSync("truncated.zstd")));

Expected: both report contiguous: true; the second just has fewer records and a lower lastSeq. No exception is raised anywhere in the read path.

Suggested fix, if you consider this worth closing. The cheapest option needs no format change and no writer change. The projection cache (<root>/session_projcache/sessions/<id>.json) already persists a per-key {ver, seq, val} fold watermark, documented as able to lag the log but never lead it (dsh-session-projection-cache/lib/index.js:203-204). So cache.seq > log.lastSeq is already a witness sitting on disk. On a 470-log store, cache.seq equals the current generation's last seq in 270/270 sessions where both exist (0 lagging, 0 leading), so the invariant is currently exact and a violation would be meaningful. Comparing them on open would surface the loss without touching the log format.

Alternative, if you would rather the file be self-describing: emit a periodic session/checkpoint event carrying the current seq and byte offset (or running event count), or declare a total when the log is sealed.

Severity, stated plainly. This is a detection gap, not a data-loss bug we can reproduce in normal operation. The write path fsyncs whole frames and rolls a partial append back to the previous size, so reaching this state requires an event that destroys already-durable data (filesystem damage, a partial restore, truncation by another tool). Across a 470-log store every log decodes cleanly; we have found no damaged log. Two further caveats worth having on the record:

• Any witness only converts silent loss into a visible error; it does not prevent the loss.
• An unfinished tail cannot itself be treated as damage. 98 of 470 logs legitimately do not end at turn/end (86 session/end-seed, 3 session/title, 3 tool/call, 2 each model/selection, approval/policy, step/start), and V4 keeps incomplete tails open by design (dsh-session-format-v3-to-v4: "incomplete tails retain their open transactions"). Resuming a short log also appends session/end-seed (dsh-session/lib/index.js:1312), so it can come back looking deliberately closed. The signal has to be an explicit recorded total, not an inferred shape.

[yamingmou]

The four properties hold on the published artifact, and one of them has a counterpart in an earlier thread.

Versions. I read @deepseek-ai/dsh-session-persistence-jsonl@0.1.7-rc.1 and @0.1.7-rc.2 for the persistence side, and @deepseek-ai/dsh-session-projection-cache@0.1.7-rc.2 for the cache you propose as the witness. Every location you cite lands where you say it does: scanZstdFrames at :1300, CHECKSUM_OPTIONS at :1291 with compressZstdFrame at :1369, the frame-header parse at :1318–:1324, observedLength declared at :36 and used at :95 and :104–:105. The two persistence releases are identical symbol for symbol — the same five occurrences of scanZstdFrames, the same nine of tornTruncateTo — so this is not a version-specific window between them.

One line is strict inside its domain and silent outside it

Your first and fourth properties are two readings of a single comparison. In readZstdPrefix the reader checks what it committed against what it read:

const complete = scanner.checkpoint();
if (complete.committedBytes !== complete.inputBytes) throw new Error("corrupt Zstandard session log: complete frame contains a torn JSONL record");

That is :2963–:2964 on rc.2 — zero tolerance inside its domain. And #5104 is the same line seen from the other side: there, 180 of 267 logs were rejected whole, history included, because one record in the last complete frame lacked its newline.

The domain is the frame list, and the frame list is supplied by the object under check. scanZstdFrames (:1300) walks frames by index with no declared total and returns tornStart only when EOF lands inside a frame header or body. A frame that is simply gone was never in the list, so it was never in scope for the comparison either. Loss of records and loss of frames therefore fail in opposite directions — one over-reported, one never reported — out of the same instruction.

I do not read that as a design accident so much as a recurring shape: when a completeness check's domain is derived from the data it is checking, strictness inside the domain carries no information about coverage of it.

What that means for the two proposals in this family

#5104 proposes to demote the torn-record case — stop throwing, truncate at the offending frame's start, keep the complete records. If that lands, the two damage classes converge in what a reader sees: a frame torn inside and a frame lost whole both arrive as a shorter session that opens without complaint.

That does not make the gap here worse; it makes it quieter. Your second caveat is the reason: an unfinished tail is legitimate (98 of 470 in your store), so shape cannot carry a completeness verdict — and if the torn case stops raising its hand either, then a recorded total is no longer one option of two but the only signal left.

The same field also says why this cannot be closed by tightening a check. In readZstdPrefix there are exactly two endings: tornStart present, and tornTruncateTo holds that offset (:2986–:2994); or tornStart absent, and tornTruncateTo is undefined (:2970–:2973). A complete file and a file missing its final frames land in the second case with the same value, and the field has no third state to give. There is nothing to assert against — a state has to be added.

A boundary to state with the cache witness, since it is the cheap one

dsh-session-projection-cache states the relation you quote at :203–:204, and its ground is write ordering: writes flush the log first, so a checkpoint can lag the log but not lead it. For damage that arrives after the checkpoint exists, that is exactly the witness needed.

What bounds it is what the cache is. The same package describes a stored record that fails schema as disposable derived data, moves it aside under invalidRecords: 'backup-and-skip', logs the failure, and serves the session as uncached — "a cold read rebuilds and rewrites it" (lib/index.js:84–:88). Its README states that no runtime companion for the invariant is published and that the correctness relation is only checkable by re-running it (README.md:93).

So the witness holds while a checkpoint predating the damage is still there to compare against, and it is gone once a cold read has rebuilt from the log itself — where both sides then agree, because both now come from the same shorter file. If the event that shortened the log also cost the cache its record, the witness and the thing witnessed disappear together. That is not an argument against using it; it is the boundary that travels with it, and it is cheap to state now and expensive to discover later.

Authorship note: code locations were read by me against the published tarballs of the versions named above; drafting assisted by AI; verification and publication by me. I have no engineering background — if any technical claim reads wrong, please call it out; I will re-verify against the toolchain and correct.

Reported by the OfferKuai Team — Founder: Zhaofeng (Yaming). Website: https://www.offerkuai.com/ | Contact: contact@offerkuai.com

---

中文版

四条性质在已发布产物上都成立，而其中一条在一个更早的帖里有一个对应物。

版本。持久化一侧我读的是 dsh-session-persistence-jsonl@0.1.7-rc.1 与 @0.1.7-rc.2；你提议作见证的缓存，读的是 dsh-session-projection-cache@0.1.7-rc.2。你引的每一处都落在你说的位置：scanZstdFrames :1300、CHECKSUM_OPTIONS :1291 与 compressZstdFrame :1369、帧头解析 :1318–:1324、observedLength 声明于 :36、用在 :95 与 :104–:105。这两个持久化版本逐符号一致 —— scanZstdFrame 各 5 处、tornTruncateTo 各 9 处 —— 所以这不是它们之间某个版本才有的窗口。

同一行：域内零容忍，域外无声

你的第 1 条与第 4 条，是同一处比较的两种读法。readZstdPrefix 里，读者拿"已提交的"对照"读到的"：

const complete = scanner.checkpoint();
if (complete.committedBytes !== complete.inputBytes) throw new Error("corrupt Zstandard session log: complete frame contains a torn JSONL record");

这在 rc.2 上是 :2963–:2964，域内零容忍。而 #5104 是同一行从另一侧看上去的样子：那份报告里，267 个日志有 180 个被整条拒收、历史一并不可读，原因只是最后一个完整帧里有一条记录缺了换行。

域就是那份帧列表，而帧列表由被检查的对象自己提供。 scanZstdFrames（:1300）按索引走帧、没有声明的总数，只有在 EOF 落进帧头或帧体时才返回 tornStart。帧如果没了，它就从未进过列表，也就从未进入那次比较的射程。 于是"丢记录"与"丢帧"朝相反方向失效 —— 一个被过度上报，一个永不上报 —— 而它们出自同一条指令。

我不把这读成设计上的偶然，而读成一种会重复出现的形状：当完整性检查的"域"由它要检查的数据自己推导时，域内的严格程度对"域覆盖了多少"不提供任何信息。

这对本族的两个提案意味着什么

#5104 的提案是把撕裂记录那一类降级：不抛、在该帧起点处截断、保留完整记录。如果它落地，两类损坏在读者眼里会合流 —— 帧内撕裂与整帧丢失，都会以一个"打开时不抱怨的更短会话"到达。

这不会让这里的缺口更严重，只会让它更安静。理由正是你的第二条 caveat：未完成的尾部是合法的（你的店里 470 个里有 98 个），所以形状承担不了完整性判定 —— 而若撕裂那一类也不再举手，那么**"显式记录总数"就不再是两个选项之一，而是仅剩的信号**。

同一个字段也说明了为什么这件事没法靠收紧检查来解决。readZstdPrefix 里只有两种收尾：有 tornStart，tornTruncateTo 就带那个偏移（:2986–:2994）；没有 tornStart，tornTruncateTo 就是 undefined（:2970–:2973）。"完整文件"与"丢了末尾若干帧"落在同一种情形、取同一个值，而这个字段没有第三种状态可给。没有东西可供断言 —— 要加的是一个状态，不是一道检查。

给缓存见证划一条随行的边界（因为它是便宜的那个）

你引的 :203–:204 那句关系，dsh-session-projection-cache 确实写着，而它的保证来源是写入顺序：先刷日志，所以检查点可以落后于日志、但不能领先它。对于在检查点存在之后才到达的损坏，这正是要的见证。

约束它的是这个缓存本身是什么。 同一个包把"不通过 schema 的已存记录"描述为可丢弃的派生数据，走 invalidRecords: 'backup-and-skip' 把它移开、记录具体校验失败、并以 uncached 方式提供该会话 —— "一次冷读会重建并重写它"（lib/index.js:84–:88）。它的 README 也写明：这条不变式没有运行时伴生物，其正确性关系只能靠重跑校验（README.md:93）。

所以：只要"损坏之前那个检查点"还在，见证就成立；一旦发生过一次从日志自身出发的冷读重建，它就没了 —— 因为那时两侧会一致，两侧都来自同一个更短的文件。若缩短日志的那个事件同时也让缓存丢了那条记录，见证与被见证的东西会一起消失。 这不是反对用它，这是必须随它一起说出口的那条边界 —— 现在说很便宜，日后发现很贵。

声明：上列代码位点由我本人对照上述版本的已发布产物重新通读；文稿撰写由 AI 辅助；核验与发布由我本人负责。我没有工程背景 —— 若任何技术表述有误，请直接指出，我会对照工具链重新核实并更正。我们是插件作者、不是维护者，本文不陈述任何官方政策。

本报告由 OfferKuai（Offer快）团队提交 —— 创始人：Zhaofeng（Yaming）。官网：https://www.offerkuai.com/ | 联系：contact@offerkuai.com

[xiaoshenming]

One caveat for the witness you propose, from a store where the naive comparison already produces a false violation — plus the reason we do not assert on it ourselves.

The comparison has to name which log it is comparing. A session directory can hold several generation files at once (session.v3.jsonl.zstd beside session.v4.jsonl.zstd), and cache.seq tracks the current generation. On our dev store one session has exactly that pair: the v3 file's last seq is 22 while cache.seq is 23 — a violation by the naive rule — and the v4 file's last seq is 23, exactly equal. Restricting the comparison to the canonical generation makes it exact.

And the relation is one-directional in practice, not just in principle. Across the 6 canonical logs in that store that have a cache record: 5 have cache.seq == lastSeq, and 1 lags by 2. So a lag is a real, observed state and any implementation has to tolerate it — cache.seq > lastSeq is the only state that carries information.

Why we do not report on it: the log is the store that gets rewritten, and legitimate rewrites shorten it. Our repair's committed-gap path truncates to the last turn/end before the gap, so after that the file is shorter than the cache watermark by construction — a check firing on cache.seq > lastSeq would fire on a file that is now considered good. We report only torn-tail (EOF inside a frame, where the frame list itself says something is missing) and leave whole-frame loss unjudged, which lands where you landed: the signal has to be an explicit recorded total, not an inferred shape. The blind spot and the witness boundary are now written down in docs/REPAIR-SPEC.md §5.