Skip to content

docs: inventory every in-process cache and correct three stale claims - #427

Merged
rmanibus merged 1 commit into
mainfrom
docs/cache-inventory
Aug 3, 2026
Merged

docs: inventory every in-process cache and correct three stale claims#427
rmanibus merged 1 commit into
mainfrom
docs/cache-inventory

Conversation

@rmanibus

@rmanibus rmanibus commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

Summary

Follow-up to #426. Answers "are any of our caches redundant with each other, and is this written down anywhere" — the answer to the first is no, and to the second was also no.

Adds docs/caching.md: every in-process cache, what it holds, its lifetime and bound, and the pairwise reasoning for why none subsumes another. The pairs worth stating explicitly:

  • KeyCacheStore vs PackStore's catalog — the one genuinely close call. Both answer "does key X exist" in memory, but the pack catalog knows only packed objects and gives only positive answers, falling through to a backend Exists on a miss. KeyCacheStore also answers negatively (a miss under a listed prefix is a definitive absence), which is the question backup's dedup check asks millions of times, and it elides the write entirely for a known content-addressed key with singleflight collapsing concurrent writes of the same key.
  • NodeStore LRU vs PackStore's pack LRU — different granularity at different layers: 4096 decoded nodes vs 4 raw 8 MB blobs. Neither keeps the other's hot set resident.
  • findScanner.evaluated vs metaLoader — deliberately inverted. find uses an uncached loader because a full scan crosses every snapshot, and memoizes the small verdict instead of the large object.

It also documents the three engine maps that look like caches in a listing but are not, so nobody unifies them with one: backup's newMetas (a staging area for filemetas whose bytes are still unwritten, so a loader would fail outright for them) and pendingMetas (a write buffer), and copy's remap tables (source ref → destination ref, not store content).

Three corrections

Found while checking the code against the docs. All three were in AGENTS.md, which is what agents read first:

  • KeyCacheStore was described as caching "in a temporary bbolt database". It uses in-memory maps — there is no bbolt anywhere in the module, nor in go.mod.
  • The store-layering diagram placed KeyCacheStore between PackStore and the backend, as part of the chain NewClient assembles. NewClient never builds one; NewKeyCacheStore has exactly one non-test call site, in BackupManager, which wraps the whole chain from above for a single run. That position is load-bearing: it is what lets a known key skip the write before compression and encryption rather than after them.
  • TransactionalStore has not existed for some time. The mechanism is hamt.Txn, which holds dirty nodes in memory and serializes only the dirty spine on Commit. docs/spec.md carried the same stale name.

Reviewer notes

  • Documentation only — no code changes. Included on this branch so the corrected layering description lands with the inventory that depends on it.
  • Cross-linked from the AGENTS.md documentation map and from docs/storage-model.md.
  • rfcs/0020-worm-mode-backup.md §89 repeats the bbolt claim ("KeyCacheStore's bbolt database is a local temporary file, not a repository object"). I left it alone, since RFCs are point-in-time design records rather than living specs. Worth a look: the conclusion still holds — in-memory is even more clearly not a repository object — but the premise is wrong, so if that reasoning is load-bearing for WORM mode it deserves a deliberate correction rather than a silent edit. rfcs/0001 and rfcs/0017 mention TransactionalStore and are left as historical record for the same reason.

Verification

  • env GOCACHE=/tmp/cloudstic-gocache go test -count=1 ./internal/apicheck passes
  • env GOCACHE=/tmp/cloudstic-gocache go build ./... passes
  • npx markdownlint-cli2 '**/*.md' — 0 issues in 40 files

Adds docs/caching.md: what each cache holds, its lifetime and bound, why no
two are redundant, and the three engine maps that look like caches but are
not (backup's newMetas staging area and pendingMetas write buffer, copy's
remap tables).

Three claims in AGENTS.md were wrong and are corrected:

- KeyCacheStore was described as using "a temporary bbolt database". It uses
  in-memory maps; there is no bbolt anywhere in the module.
- The store-layering diagram placed KeyCacheStore between PackStore and the
  backend, as part of the chain NewClient assembles. NewClient never builds
  one. BackupManager wraps the whole chain in one from above, for a single
  run, which is what lets it skip a write before compression and encryption
  rather than after them.
- TransactionalStore has not existed for some time; the mechanism is
  hamt.Txn, which holds dirty nodes in memory and serializes only the dirty
  spine on Commit. docs/spec.md carried the same stale name.

Cross-linked from the AGENTS.md documentation map and docs/storage-model.md.
@rmanibus rmanibus added documentation Improvements or additions to documentation area/core Core backup engine, repository model, and restore semantics labels Aug 3, 2026
@rmanibus
rmanibus merged commit 9075be6 into main Aug 3, 2026
5 checks passed
@rmanibus
rmanibus deleted the docs/cache-inventory branch August 3, 2026 20:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/core Core backup engine, repository model, and restore semantics documentation Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant