[Perf] One list() call walks the session corpus twice: historicalCorpusRevision() re-runs listGenerations()

[uWydnA]

Summary

SessionPersistence.list() enumerates the whole session corpus twice per call, through the same listGenerations() function. The second enumeration additionally stats every selected path and folds it into a sha256.

This is a separate, self-contained defect from the per-session header scan analysed in #7142. That thread's fix (bounded concurrency + identity-keyed caches) makes each walk cheaper; this one can be removed outright, because both consumers run inside the same call that already holds a corpus snapshot.

I am filing it separately because I did not see it mentioned in #7142, #5043, #3740, #4285 or #5961, and because it is small and independent enough to be worth its own line item. If you would rather track it inside #7142, that is fine by me — a companion comment with the same numbers is already there.

Environment

• @deepseek-ai/dsh@0.1.7-rc.2 (npm), macOS arm64, Node v22.22.0, web profile on loopback
• session-persistence-jsonl at defaults (compression: zstd); session-query-sqlite at path: ':memory:', openAt: never
• Store: 816 sessions / 654 MB (1,361 sessions / 1.35 GB before I pruned it)

The double walk

// packages/session/session-persistence-jsonl/src/index.ts — master, lines 483-485
const artifacts = await this.listArtifacts(signal)                         // :1064 -> listGenerations()
const corpusRevision = artifacts.some(a => a.sourceVersion < SESSION_FORMAT_VERSION)
  ? await this.historicalCorpusRevision(signal) : undefined                // :1039 -> listGenerations()

listGenerations() (:1025) is the full two-level walk — listProjectDirs() → listSessionDirs() → resolveGenerationInDirectory() per directory. Both callers invoke it independently, so one list() runs the identical walk twice; the second one then adds a stat() per selected path plus a sha256 over all of them.

The trigger only needs one pre-current-format artifact, so in practice it is essentially always taken: on this store 746 of 816 selected generations (91.4%) are < v4 (v0 152, v3 594, v4 70); on the larger pre-prune store, 1,291 of 1,361.

Measured

Warm, median of 5, 816 sessions / 654 MB:

term	ms
listGenerations() equivalent (two-level readdir + generation resolve)	87
historicalCorpusRevision() equivalent (same walk + stat + sha256)	97

So roughly 90% of the corpus-revision term is a duplicate of work the same call has already done.

For scale, in the same measurement series a full cold listing on the 1,361-session store decomposed as: readdir pass 3.01 s, header reads (open + 8 KB + zstd) 7.24 s, corpus-revision 1.78 s — sum 12.2 s, against the endpoint measured cold at 11.6 s and warm at ~1.5 s. So this term is second-order next to the header scan that #7142 is about; I am not claiming it explains the slowness, only that it is free to remove. (Cold/warm really is the dominant variable here, not CPU: a single open + 8 KB read is p50 0.38 ms, p99 1.91 ms.)

Suggested change

Hoist the single listGenerations() result in list() and pass it to both consumers instead of letting each call it. The call already holds a snapshot of the corpus, so sharing one enumeration is at least as correct as today — and strictly more consistent, since the two walks currently run at different instants and are free to disagree with each other.

Notes

• Still unfixed on master (latest push 2026-09-24, fetched 2026-09-27): listArtifacts() at line 1056 is still the serial nested loop with one await per session — no per-instance cache, no bounded concurrency — and rootEncodingCheck (:1608) is still the only memo on the path. The 8.1x patch reported in [Perf] session/list is O(sessions) with no summary cache: 2.5-3.0 s and 59.8 MB of reads for ~7,700 sessions #7142 has not landed.
• This does not replace [Perf] session/list is O(sessions) with no summary cache: 2.5-3.0 s and 59.8 MB of reads for ~7,700 sessions #7142. The dominant cost there — one serialized header read per session, re-done on every call — is untouched by this change.
• Why not just migrate the store to the current format. Migration clears the some() trigger, but it also leaves both artifacts in the directory for the walk to enumerate, so it moves two variables at once. That is consistent with the negative result in [Perf] session/list is O(sessions) with no summary cache: 2.5-3.0 s and 59.8 MB of reads for ~7,700 sessions #7142 and is why I would not use migration to price this term.
• User-visible impact today is on the same two endpoints as [Perf] session/list is O(sessions) with no summary cache: 2.5-3.0 s and 59.8 MB of reads for ~7,700 sessions #7142: session/list and sessionReferenceResolver/candidates (the @ mention menu). Pruning the store remains the only effective lever in the field for now — moving 40% of it out took session/list from 1,490–2,349 ms to 428–533 ms and candidates from 1,487–2,720 ms to 432–834 ms here.