Framing audit: per-frame costs dominate session cold-open — measured 15× decompress / 2.7× storage from frame merging (B-stage evidence for #4416)

[MedicineKing]

Framing audit: per-frame costs dominate session cold-open — measured 15× decompress / 2.7× storage from frame merging (B-stage evidence for #4416)

Standalone measurement post, not a duplicate of #4416: that thread identified the direction (B-stage: merge write frames); this post supplies the missing measured magnitude — on real artifacts, with the official reader, no source surgery.

The observation

Session logs are written one zstd frame per append batch (encodeEventBatch → one compressZstdFrame per call). Early batches are small, so large sessions accumulate tens of thousands of frames — d1bf7081 on my machine: 42,354 frames in 12.7 MB = ~300 compressed bytes per frame. Every frame carries fixed framework overhead — zstd magic (4 B) + frame header (2-14 B) + block headers + checksum trailer (4 B), roughly 10-20 B/frame — that gets amortized over ~300 B of payload, and small frames also compress worse. I wanted the number.

Method

4 largest sessions from a 19-session library, decoded through the official rc.2 path (published npm 0.1.1-rc.2, decodeStorageRecord, unmodified; vendored from npm). Stages timed: file read+frame scan, per-frame zstd decompress, line split + JSON.parse, row decode to events. No content leaves the machine — session IDs shortened, numbers only.

Results

session	size	frames	events	decompress	parse	decode	decode-path total
d1bf…	12.7 MB	42,354	657,284	558 ms	98 ms	38 ms	700 ms
1e08…	8.6 MB	29,585	465,641	406 ms	44 ms	22 ms	474 ms
eb5f…	9.1 MB	17,531	832,692	253 ms	62 ms	30 ms	348 ms
6289…	9.6 MB	25,669	536,655	350 ms	54 ms	29 ms	434 ms

Finding 1 — decompression is ~80% of the decode path (558/700 through 253/348); JSON.parse + per-row decode combined are 8-20% each. Optimizations that skip bytes (seek/R2) or frames (merge/B) target the dominant term; optimize-parse targets a minor one.

Finding 2 — frame-merge magnitude (the part #4416's B-stage estimate did not have). Same text, re-framed the way a merged writer would (header line alone in frame 1, events in the rest): decompress 583 ms → 39 ms (14.9×) — a fresh single cold run, slightly above the table's median-of-5 baseline (558 ms; using that baseline gives 14.3×) — and file 12.7 MB → 4.7 MB (2.7×). The missing bytes split into two distinct terms: fixed frame overhead (magic + header + checksum ≈ 10-20 B/frame × 42,354 frames ≈ 3-7% of stored bytes — small), and small-frame compression inefficiency (the remaining ~56-60%: each frame compresses in isolation, so there are no cross-frame matches and the history window is tiny — a 300 B payload cannot exploit the log's redundancy). Both terms scale with frame count (42,354 → 2), so frame-reduction wins on both axes at once: fewer frames → less fixed overhead and much better compression. Scope note: the merge measurement is the single largest sample (d1bf); full-corpus before/after numbers are pending an actual B-stage branch — happy to run them through the #4945 gate when one lands.

Relation to the roadmap

• B-stage (merge write frames) is the direct lever for both numbers — the roadmap's "reduce frame count 10-50×" conjecture (verified against DSH: Why Opening a Large Cold History Session Can Stall the Whole Web Server — A Systematic Root-Cause Analysis and Optimization Roadmap #4416's text) is now quantified: frame-reduction delivers −63% storage and −93% decompress time at the extreme (2 frames), and the curve is smooth (fewer/larger batches → less fixed overhead and better compression), so even a 10× frame reduction captures most of it.
• A-stage (tail index sidecar + seek) interacts: smaller frame counts make per-frame offsets coarser but cheaper to index — complementary, not competing.
• Both stages change the physical layer — exactly the thing our #4945 corpus gate referee (assertZstdHeaderFrame / Session.fromRestore / scanner contract, official reader) was built to check. I can run that gate on any candidate branch merging at either stage; the expectation table (partial-tail → ACCEPT trimmed, two-writer → REJECT) is already published.

Reproducibility

• Rig: 19-session library (local), stages timed with performance.now(), medians of 5 where relevant; script is 100 lines, no dependencies, no content in output — happy to share the measurement(s) and to re-run against a candidate branch.
• Caveat: numbers are decode-path only (no UI/surface folding/projection — those are DSH: Why Opening a Large Cold History Session Can Stall the Whole Web Server — A Systematic Root-Cause Analysis and Optimization Roadmap #4416's R3/R4 territory); cold-open end-to-end is dominated by this path plus those, and the split is what makes both measurement sets complementary.

[denial123789]

I turned the measured framing result into a reusable operator guide: https://github.com/sandbaseai/deepseek-harness-handbook/blob/main/docs/en/architecture/session-framing-performance.md. It keeps decode-path timing separate from UI cold-open claims, preserves the official reader/corruption-corpus gate, and records the zstd level tradeoff from #4948. The handbook is independent community documentation, not an upstream implementation claim.