[Perf][P0] open() on every pre-V4 session unconditionally scans the entire session corpus via listArtifacts() — O(N) per open, p50 1167 ms at N≈1830

[edisonzerolam]

Environment

Item	Value
Core package	@deepseek-ai/dsh-session-persistence-jsonl@0.2.0-rc.2 (other core packages on the same line)
Desktop shell	dsh-plugin-desktop@2.0.17 (DSH Desktop)
OS	Windows 11 Pro Insider Preview, build 26220, x64
Session corpus	22 workspaces / 1833 sessions / ~1.1 GB
Generation mix at audit time	v0 538 · v3 865 · v4 428 — only 23.4% are V4

Where this report comes from: I ran a performance audit on one real machine. The numbers below are all measured, not estimated. I did not modify any product source code; the "after" numbers come from a data-side workaround described at the end.

Summary

Opening one historical session costs about 1.2 s, and the cost is O(total number of sessions in the corpus) — it is not a function of the session being opened. On this machine p50 = 1167.3 ms over 1831 sessions.

The cause: for any session stored below generation 4, prepareStoredMigration() calls listArtifacts() unconditionally, which walks every session directory in the corpus and reads the first line of every zstd log, purely to answer "does this session have subagent children?". For 91.9% of sessions the answer is empty. The read path never persists the migration result, so the work repeats on every open and after every restart.

After pre-migrating the corpus to V4 — no source changes — the same measurement drops to p50 = 9.9 ms (≈118×).

Root cause

packages/session/session-persistence-jsonl/lib/index.js

// :2680  prepareStoredMigration(id, selected, signal)
// :2684
const children = async () => (await this.listArtifacts(signal)).filter(
  (source) => source.header.origin === "subagent" && source.header.parentSession === id);

listArtifacts() (:3031) = ensureRootEncoding() + listGenerations() + one readGenerationHeader() per artifact:

// :3004  listGenerations —— full corpus directory walk
for (const project of await this.listProjectDirs(signal))
  for (const dir of await this.listSessionDirs(project, signal)) { ... resolveGenerationInDirectory(dir) }
// :3041  —— then read the zstd first line of every one of them
header = await this.readGenerationHeader(selected, void 0, signal);

Three facts make this pathological:

1. It is unconditional. It runs even for sessions that have no children and never can.
2. The read path does not persist the result. publishStoredMigration() is called only from the write branch (:2495); the read branch (:2473–:2484) only sets state.primed. So every open(id, "read") redoes the full migration, and the corpus keeps staying pre-V4.
3. The in-process memo barely helps. COLD_LOG_MEMO_MAX_ENTRIES = 2 (:2284); it is an instance field (lost on restart); loadStoredMigration only hits it when status === "prepared" (:2652). Two concurrently open sessions → two independent full scans with no sharing (migrationPreparations is keyed per id).

Per dsh-session-format-v3-to-v4/README.zh.md:84 — 「当前 V4 打开跳过发现」 — V4 embeds the parent catalog in the log, so V4 sessions already bypass this entire path. That is exactly why pre-migrating to V4 fixes it without touching a line of product code.

Evidence

(a) Phase instrumentation — open(id, "read") on a fresh instance, n=3 average, comparing a 1-session mini corpus with the real 1833-session corpus:

phase	mini (1 session)	full (1833 sessions)	delta
ensureRootEncoding	0.9 ms	103.2 ms	102.3
findLog	0.9 ms	4.7 ms	3.8
listArtifacts	2.2 ms	688.0 ms	685.8
requireStoredLog (total)	7.7 ms	1288.5 ms	1280.8

findLog is 4.7 ms — it is not the bottleneck. The corpus walk is.

(b) Isolation experiment — monkey-patching be.listArtifacts = async () => [] in-process (zero disk changes), 8 v0/v3 samples:

native                    : 10174.7 ms   (1271.8 ms/session)
listArtifacts disabled    :   935.5 ms   ( 116.9 ms/session)
saved                     :  9239.2 ms   (90.8%)

One sample: 1253.4 ms → 112.7 ms.

(c) The scan is almost always useless — reading only the first line of all 1833 logs:

origin === "subagent"                : 1422 (77.6%)
has a parentSession field            : 1452 (79.2%)
sessions that actually HAVE children :  148 ( 8.1%)
→ 91.9% of opens scan the whole corpus and get []

(d) Corpus-size scaling — synthetic corpora of N sessions, open p50:
N=1 → 1.2 ms · 250 → 21.3 · 500 → 43.7 · 1000 → 66.3 · 1500 → 139.6 · 1833 → 125.6
normalised 6.6 – 10.3 ms per 100 sessions: strictly linear.

(e) Before/after on the real machine — 1831 sessions, real open(), no source changes:

	before	after V4 pre-migration
total wall time	1640.5 s	56.2 s
open p50	1167.3 ms	9.9 ms
p90 / p99	1590.1 / 1995.1 ms	25.5 / 193.3 ms

Verified across two independent application restarts (56.2 s → 56.5 s; p50 9.9 → 9.9 ms), so it is a durable on-disk change, not a warm-cache artifact.

Reproduction

// <APP> = <install dir>/resources/app/node_modules/@deepseek-ai/
import { resolve } from "node:path";
const APP = "<APP>";
const { default: JsonlSessionPersistence } = await import("file:///" + APP + "dsh-session-persistence-jsonl/lib/index.js");
const { Context } = await import("file:///" + APP + "cordis/lib/index.js");
const rootCtx = new Context();
rootCtx.provide("logger", { warn(){}, info(){}, debug(){}, error(){} });
const be = new JsonlSessionPersistence(rootCtx.isolate([]),
  { root: resolve("<HOME>/.dsh/sessions"), compression: "zstd" });

for (const id of ["<any v0 or v3 session id>"]) {
  const t = performance.now();
  const h = await be.open(id, "read");      // ← slow
  console.log("native ", (performance.now() - t).toFixed(1), "ms");
  await h.close?.();
}

be.listArtifacts = async () => [];          // ← disable only the corpus walk
for (const id of ["<the same id>"]) {
  const t = performance.now();
  const h = await be.open(id, "read");
  console.log("patched", (performance.now() - t).toFixed(1), "ms");
  await h.close?.();
}

Scaling variant: build mini corpora of N = 1 / 250 / 500 / 1000 / 1833 sessions and plot open p50 — it is linear in N, which alone proves the cost is corpus-bound rather than session-bound.

Suggested fix

Disclosure: these suggestions are code-level inferences. I did NOT patch the product source and measure the result. What I verified end-to-end is the data-side workaround: pre-migrating the corpus to V4 using DSH's own open(id, "write") (which triggers publishStoredMigration). That is what produced the 118× number above. The changes below are proposed from reading the call graph, not from a measured patch.

In rough priority order:

1. Skip the discovery when the session cannot have children. The origin === "subagent" filter means only sessions that ever spawned subagents matter. Maintain an origin → parentSession index (the header already carries both fields) instead of walking the corpus. Expected: removes ~90% of the cost for every pre-V4 open. Risk: the index must be invalidated on create/delete — same lifecycle as ensureRootEncoding.
2. Persist the migration on the read path. Today publishStoredMigration() is write-only (:2495). Caching the V4 result on read (behind a "safe to materialise" check) would make the cost one-time per session instead of per open. Risk: read opens must not silently become writers — needs an explicit, auditable opt-in. This is the change most likely to have side effects.
3. Fast path: let the caller declare "I have no children". Many call sites know this statically. Risk: a wrong declaration silently drops children from the catalog — needs a safety net.
4. Raise or rethink COLD_LOG_MEMO_MAX_ENTRIES = 2. With two sessions open (a common case) the memo thrashes by construction. Cheap change, modest gain, low risk.

Not recommended: raising the V4 cutoff without (1) — it shifts the cost rather than removing it.

Related

This is purely a performance issue. A separate family of problems (old-format sessions being refused outright) exists in the same package and is reported separately, since the causes and the fixes are unrelated.

[slyyr]

One improvement I'd consider is separating subagent-child discovery from the normal session-opening path.

Currently, if listArtifacts() scans the entire corpus just to determine whether one session has child sessions, the cost grows with the total number of sessions rather than the session being opened.

A few possible improvements:

1. Maintain a parent-child index: Store the relationship between parent sessions and subagent sessions when they are created, so opening a session doesn't require scanning every artifact.
2. Persist migration results: If the migration result is safe to cache, avoid repeating the same discovery work on every read. However, this should preserve the intended read-only behavior of open(id, "read").
3. Add a fast path: Skip child discovery when the session metadata proves that it cannot have spawned children.
4. Test scaling: Benchmark cold and warm opens with increasing corpus sizes, including concurrent opens and application restarts.

I'd prioritize the index or a safe fast path before simply increasing the cache size, since a larger cache may reduce repeated work without eliminating the underlying O(N) scan.

The key is to preserve migration correctness while making the cost of opening a session depend on that session's actual data, not the size of the entire corpus.

Hope this helps! If this addresses the issue, would you consider marking it as the accepted answer?

[edisonzerolam]

Thanks for reading and for the suggestions — they line up with the four in the post, and I agree with your priority ordering: an index or a safe fast path removes the scan, whereas a larger cache only reduces repeated work.

On marking this as the accepted answer: I'd rather not, and the reason is worth stating so it doesn't read as dismissiveness. Nothing has been fixed yet. origin/master (d743267388 = dsh-v0.2.1-alpha.2) still contains the unconditional listArtifacts() call in prepareStoredMigration, and the read path still does not persist the migration result — so the O(N) cost is live for anyone opening a pre-V4 session. Marking an answer here would tell the next reader the problem is solved when it is not. I'd rather leave it open until either upstream lands a change or someone measures a patch.

If you want to make it a real answer, the highest-value thing would be a measurement rather than a design: patch one of the four candidates, run the phase instrumentation before/after, and report the delta. The numbers I have are from a data-side workaround (pre-migrating the corpus to V4, which skips the discovery path entirely — session-format-v3-to-v4/README.zh.md:84, "V4 opens skip discovery"), not from a source patch, so the four suggested fixes remain unmeasured. A single measured candidate would settle more than four plausible ones.

Two things that might help if you do:

• findLog is not the bottleneck — it measures 4.7 ms on a 1833-session corpus. The corpus walk is (listArtifacts 688 ms of a 1288 ms open). It's easy to instrument the wrong function.
• Corpus size is the variable, not session size. N=1 → 1.2 ms, N=1500 → 139.6 ms, strictly linear at 6.6–10.3 ms per 100 sessions. A benchmark that varies session size instead of corpus size will show nothing.

Happy to compare notes if you benchmark any of them.