Migrated session logs put the entire history in a single Zstandard frame, so opening one large session allocates ~120 MB at once

[deepsleepAquarium]

Migrated session logs put the entire history in a single Zstandard frame, so opening one large session allocates ~120 MB at once.

Summary

After a session log is migrated to the current generation (session.v4.jsonl.zstd), the resulting file typically contains only a handful of Zstandard frames, and one frame holds ~100% of the plaintext. Opening such a session therefore performs one very large single allocation (and one very large JSON parse burst) instead of streaming many small frames.

The package README documents the intended layout as a concatenation of independent frames — "one checksummed frame containing only the header line, followed by each persisted batch" — so a single frame covering the whole history looks like a migration-side flush behaviour rather than the documented steady state.

On a low-spec host (4 logical cores, 7.5 GB RAM) opening two or three such sessions is enough to saturate the machine: the process sits at roughly 75% of one core continuously and the whole system becomes unresponsive until the session is closed.

Measured evidence

Frame statistics were obtained by scanning frame magic (0x28 B5 2F FD) and decompressing each frame independently (Node 24 zlib.zstdDecompressSync decodes only the first frame, so a per-frame scanner is required).

file	generation	file size	frames	largest decompressed frame
migrated session A	v4	47.64 MB	3	118.63 MB (100% of plaintext, 28,728 events)
migrated session B	v4	21.88 MB	3	119.29 MB
several other migrated sessions	v4	1.83 / 4.36 / 5.36 / 16.42 MB	3	10.47 / 23.89 / 31.20 / 87.06 MB
non-migrated session (written by the older CLI, still v3)	v3	38.80 MB	15,397	0.16 MB

Structural scan over all logs present on the host: 159 v4 files / 152.03 MB, and the sum of each file's largest frame is 108.86 MB — i.e. 72% of all v4 bytes sit inside per-file single giant frames. Every v4 file scanned had 3 frames; 16 session directories contain both a v3 and a v4 log, which is consistent with these v4 files being migration output.

Cost of opening the largest one (measured with process.memoryUsage() around the decode and parse):

• decompressing that one frame: RSS 126 MB -> 412 MB (peak arrayBuffers ~295 MB)
• JSON.parse of the 28,728 events in that frame: heap 30.6 MB -> 297.4 MB, RSS peak 563.6 MB
• i.e. a single "open" costs roughly +267 MB heap / ~560 MB RSS

Suspected cause

• packages/session/session-persistence-jsonl/src/generation.ts (decodeCurrentGeneration) reads the whole file into one Buffer and decodes frames from it; src/zstd-private-decoder.ts accumulates outputs above DECODE_CHUNK_SIZE (1 MB) via fullChunks.push(...) + Buffer.concat(...), so a 118 MB frame necessarily becomes one >=119 MB contiguous allocation before JSON parsing starts.
• The migration path appears to append the converted history as a single batch, which would produce exactly the "header frame + one giant frame" shape observed. Files produced by normal appends (older generation, or presumably new appends to a current-generation file) are many small frames.

Impact

• Any host with limited RAM can be pushed into swap/unresponsiveness simply by opening a large migrated session; the same session is cheap in a many-small-frames file.
• Users have no way to tell beforehand which sessions are in the expensive shape.

Possible directions

1. Make the migration path emit frames per persisted batch (or chunk frames to a bounded target size), matching the documented "independent frames" layout.
2. Make decoding stream frame-by-frame without materialising the whole file buffer and without a single Buffer.concat of an arbitrarily large frame.
3. Optionally expose a maintenance action to re-encode a log into bounded frames.

Environment

• DeepSeek Harness CLI checkout at release 0.1.5-alpha.2 (its own logs are the many-small-frames v3 shape and are not affected).
• Observed on current-generation (v4) logs produced by the desktop application on Windows.
• Node v24.21.0.

---

Published by @deepsleepAquarium

[deepsleepAquarium]

Additional evidence, from reading the shipped packages of this release (no host-specific data):

1. The documented layout is "one frame per persisted append batch".

@deepseek-ai/dsh-session-persistence-jsonl README says the default artifact is a concatenation of independent Zstandard frames: one checksummed frame containing only the header line, followed by one frame per persisted append batch.

So the number of frames in a file equals the number of persisted batches. A file with 3 frames is therefore a file that was written in 3 batches — which matches what we observe: header frame, one frame holding ~100% of the plaintext, plus a small tail.

2. The batching window is a per-handle scheduling policy, not a configuration option.

The same README states that write batching for live events is not configurable: the batching window is a scheduling policy inside each write handle.

3. Conclusion: this looks like an interaction, not a deliberate design.

A migration converts the entire history through a single write handle in one burst. Because the batching window is time-based, everything produced during that burst is very likely to be buffered into one batch — and that single batch becomes a single, enormous frame. Slow/streamed appends in normal operation produce many small frames instead, which is why only migrated files show this shape.

Possible mitigations (in addition to the ones in the original post):

• Bound a batch by event count and/or uncompressed bytes as well as by time, so a burst cannot produce a multi-hundred-megabyte frame.
• Force a flush (i.e. a batch boundary) at a bounded interval/event count while a migration is writing.
• Alternatively, keep the current batching policy but make the decode side stream frame-by-frame instead of materialising one frame of arbitrary size.

[deepsleepAquarium]

Root cause found in the shipped code (0.2.0-rc.2) — the migration streams its input but never forms a batch boundary on output.

The migration loop (lib/index.js, consumeMigrationBytes):

async function consumeMigrationBytes(rows, chunks, signal) {
  let yieldDeadline = performance.now() + MIGRATION_DECODE_YIELD_INTERVAL_MS;   // 500 ms
  for (const bytes of chunks) for (let offset = 0; offset < bytes.length; offset += MIGRATION_WORK_CHUNK_BYTES) { // 1 MB
    rows.write(bytes.subarray(offset, offset + MIGRATION_WORK_CHUNK_BYTES));
    if (performance.now() < yieldDeadline) continue;
    await scheduler.yield();
    signal?.throwIfAborted();
    yieldDeadline = performance.now() + MIGRATION_DECODE_YIELD_INTERVAL_MS;
  }
}

Relevant constants (same file):

const MIGRATION_DECODE_YIELD_INTERVAL_MS = 500;
const MIGRATION_WORK_CHUNK_BYTES = 1024 * 1024;        // 1 MB
const MIGRATION_WRITE_CHUNK_BYTES = 4 * 1024 * 1024;   // 4 MB

There is no flush() and no publish call anywhere in this function (checked: zero hits for flush/publish/append in the whole function body). The migration therefore only feeds the writer; it never asks the write handle to close a batch.

Why that produces one giant frame

• The artifact layout is one frame per persisted append batch (see previous comment).
• Batches are produced by a time-based batching window owned by the write handle, and draining can be paused (drainPaused is checked in the buffering path).
• The migration runs as one burst (it yields to the event loop every 500 ms, but it does not force durability), so the converted history accumulates in the write handle and is persisted as essentially one batch → one frame.

This also explains the observed shape exactly: header frame + one frame holding ~100% of the plaintext + a small tail.

Why it is invisible in normal operation

Live appends arrive slowly, so each batching window flushes a small batch and produces a small frame (a 317 KB non-migrated log has 192 frames). Only migration output — a whole history in one burst — becomes a single frame of unbounded size.

Downstream impact (why this matters beyond disk size)

Because decode is synchronous (zstdDecompressSync for the frame, then JSON.parse for its events), opening such a session blocks the host event loop for seconds (measured: one 118.63 MB frame holding 28,728 events; RSS peak 563.6 MB). On a small host this stalls the host long enough that clients lose the connection and reconnect — and reconnecting triggers another read/rebuild, which can repeat.

Suggested fix

• Force a batch boundary (i.e. flush) at each MIGRATION_WRITE_CHUNK_BYTES write during migration, so migrated files contain frames bounded by the write chunk size rather than by the whole history.
• Keep the current streaming/yielding behaviour otherwise; the input side is already well behaved.

[deepsleepAquarium]

Confirmed this is still present in the newest release, 0.2.1-alpha.2 — so it is not a fixed-in-newer-versions issue.

Checked @deepseek-ai/dsh-session-persistence-jsonl@0.2.1-alpha.2 (lib/index.js, 140 KB): consumeMigrationBytes is byte-for-byte the same logic as in 0.2.0-rc.2 — it calls rows.write(...) in 1 MB work chunks, yields to the event loop every 500 ms, and contains no flush / publish call at all (grep for flush|publish inside the function body: 0 hits).

Also for the record, the release notes do describe the intent: dsh-v0.1.7-alpha.1 says "Session logs upgraded to V4 … add a batch migration tool for developers, while preserving the originals" — the batch nature of that tool is exactly what produces the single-frame output described above.

And dsh-v0.2.1-alpha.2 mentions "Add migration guidance to sidebar forks and subagent catalog browsing, directing users to open the original session when migration is required" — i.e. migration-related UX is being worked on, but the frame-boundary behaviour itself is unchanged.

So as of 0.2.1-alpha.2, opening any session that was migrated with this path still costs one unbounded contiguous allocation, and (on small hosts) a multi-second synchronous stall.

Additional observation that may narrow the fix: the migration does bound its working set on the input side (MIGRATION_WORK_CHUNK_BYTES = 1 MB) and does yield (MIGRATION_DECODE_YIELD_INTERVAL_MS = 500 ms). Only the output side has no bound. Forcing a batch boundary at each MIGRATION_WRITE_CHUNK_BYTES (= 4 MB) write would make migrated files' frames bounded by 4 MB, which would keep decode cost proportional and bounded.