perf(cache): shard the session cache by provider and month, load only the months a query needs - #1007
Merged
Merged
Conversation
A provider's shard held its whole history, so one appended session rewrote 95 MB. Each provider's files are now split by the UTC month of their first turn - a bucket that is stable across appends, so a growing session never migrates shards - and every shard records the newest month it holds so a ranged load can skip the ones that cannot contribute. Dirty tracking is per bucket: markCacheDirty takes an optional file path and marks both the bucket the entry was last saved in and the one it is in now. A save writes only dirty buckets, carries the refs of months it never loaded, and merges the on-disk shard back in when a bucket is dirty but was never loaded. v8 and v7 caches re-lay-out losslessly.
…uery Every per-file markCacheDirty call site now names the file, so a parse, a re-parse, a failure marker, an orphan eviction and the durable age-out each dirty exactly the month they touched. The two section-level marks (a fingerprint reset, the durable stamp) stay provider-wide. parseAllSessions derives a month scope from its dateRange and threads it through every loadCache call, so a today/week query stops reading the months it cannot report on.
A file's shard span was read off turns[0]/turns[-1], but several providers emit turns non-chronologically (cursor by ROWID, goose/crush/copilot by a DESC ordering). That produced until < bucket - an empty span, so the shard was unreachable at every scope and its sessions re-parsed every run. cacheFileSpan now takes the min and max month over all turns. An entry re-bucketing out of a month the run never loaded (a re-parse that moved its oldest turn, or the #441 failure marker that has no turns at all) left the old copy in the carried shard, so one path lived in two shards and a later load could resolve to the stale one. A save now prunes those paths from the shards it carries, and a load merges shards in envelope order, resolving any duplicate to the freshest fingerprint and dirtying both buckets so the next save retires the loser. A carried month whose shard another writer had republished was dropped from the envelope outright, losing expired-transcript PR orphans no re-parse can recover. The envelope is now re-read just before publishing and the current shard name adopted; a ref is dropped only when that envelope lacks it too. The same re-read moves every merge read after the ownership fence and gives the merge one optimistic retry, so the read-modify-write window shrinks to the publish itself. Also: retire an orphaned v8 directory / v7 file left by an interrupted re-layout, age-guarded, once a v9 envelope is published.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
After #1005 a warm launch still rewrote the whole 95 MB Claude shard when one session appended, and every query loaded and parsed all ~150 MB of shards even for
--period today(28% of a warm run).What
Provider × month shards (
session-cache.v9/). A file's bucket is the min month over its turns (stable across appends), with anuntil= max month, so a long-lived session started months ago is still found by a today-scoped load. Turn-less files (failure markers) live in an always-loaded0000-00bucket. Provider metadata moves into the envelope so a partial load can build a correct section.markCacheDirty(cache, provider, filePath)dirties both the bucket an entry was last saved in and its current one.Scoped, concurrent load.
loadCache(scope)reads only the months a query's date range needs (± one month below for branch/PR carries and spawn anchors), concurrently, merged in envelope order. Never scoped: durable providers (the cache is the only record of pruned usage) and providers whose env fingerprint changed. Save carries unloaded months' envelope refs verbatim; a dirty-but-unloaded bucket merges on save.Integrity (independent adversarial review found and we fixed): out-of-order turns previously produced an empty span → unreachable shard; a re-bucketed entry under partial load could survive in two shards (write side deletes moved paths from other shards; read side resolves duplicates by freshest mtime and dirties both); a carried month republished by another writer was dropped from the envelope (now re-reads the envelope pre-publish and adopts the current name); merge-on-save reads moved after the fence + envelope re-read with one optimistic retry (residual window is the envelope rename; product saves are serialised by the refresh lock; documented). Orphaned v7/v8 layouts are swept once a v9 envelope is published.
Migration v8→v9 and v7→v9 lossless, in memory, no re-parse (~0.5 s on a real 150 MB cache).
Measured (real 6 GB corpus,
--period today, one appended record per warm run)Tests
tests/session-cache-shards.test.ts8 → 24: bucket stability across appends,untilbookkeeping, out-of-order turns reachable at both scopes, per-month dirty tracking (untouched months byte-identical), delete dirties the old month, scoped load reads only in-scope months, scoped save preserves unloaded months byte-for-byte, merge-on-save, re-bucketing out of an unloaded month (re-parse and failure-marker cases → exactly one copy), duplicate resolution + prune, carried months under a concurrent writer (orphans survive), durable/fingerprint providers never scoped, v8→v9 / v7→v9 lossless, month-level corrupt isolation + self-heal, prior-layout sweep. 7 of the 8 new integrity tests fail on the pre-fix code.tscclean ·npm test2700 passed ·test:locks26 · app 488.Note (not fixed here):
codex-results.json(58 MB) and the codex shards (53 MB) are derived layers of the same usage — the former carries the #1005 resume anchors. Collapsing them is a separate change.