perf(engine): stop retaining filemetas nothing reads

[rmanibus]

Summary

#429 proposed bounding the filemeta loader with an LRU, the way NodeStore bounds its node cache. I benchmarked that first — it would have been a regression. The measurement pointed at two better targets, and this PR does those instead.

Why the LRU doesn't work

A filemeta traversal sweeps each ref exactly once per snapshot, uniformly. Under a cyclic sweep an LRU evicts precisely the entry the next sweep is about to ask for, so the hit rate doesn't degrade — it collapses. From BenchmarkMetaLoaderDiffPattern, eight sweeps:

working set	cache 1024	cache 4096	cache 16384
1,000 files	87.5%	87.5%	87.5%
10,000 files	0.0%	0.0%	87.5%

At 10k files a 4096-entry cache turns 10,000 store reads into 80,000 — 8× the API calls on a remote backend. NodeStore is bounded successfully because a HAMT descent re-touches the root and upper levels on every lookup; a filemeta sweep has no equivalent locality. Any fixed bound has this cliff, it just moves. So the cache stays unbounded and the memory came out of the traversals instead.

prune memoized filemetas it could never re-read

markFileMeta returns early for a ref already in its reachable set, so a load happens at most once per run however many snapshots share the tree. Measured over four snapshots of a 2000-file tree: the cache ended with 2000 entries and zero hits. It now reads through.

files	retained before	after
5,000	3.3 MB	1.3 MB
20,000	12.7 MB	2.9 MB
50,000	28.1 MB	4.0 MB
100,000	54.6 MB	6.7 MB

diff retained every entry in a folder-only lookup

collectMetadata's map is read at exactly one place — byID[parentID] in toFileChange — and a parent is a folder. Retaining files there held a core.FileMeta each for entries nothing read.

files	parent entries before	after	peak before	peak after
5,000	10,504	504	11.7 MB	8.6 MB
100,000	210,004	10,004	275.5 MB	217.6 MB

An entry naming a non-folder parent is unaffected: collectMetaPaths already ends a chain at the deepest ancestor it can resolve, which is its documented behaviour for any unresolvable parent.

Neither change costs a single store read — that is what separates them from bounding the cache.

Closes #429

What's added

• metaloader_bench_test.go — the two benchmarks that produced these numbers: retained bytes vs file count, and hit rate under the diff access pattern.
• metaloader_memory_test.go — tests pinning both choices. Both regressions are silent, which is why they need pinning: a cache that stops being consulted still works, and one that starts retaining everything still returns the right answer. TestDiffRetentionGrowsWithFoldersNotFiles asserts on entry counts rather than heap, which is too noisy to assert on.
• TestDiffReportsFullPathsAfterFiltering proves the filter doesn't shorten a nested path — the point being that the dropped entries were unread.

Reviewer notes

• docs/caching.md is corrected. It claimed no cache was redundant. prune's was, and I only found it by counting hits rather than reading the code — which is why the benchmarks now exist. The doc says so explicitly rather than quietly editing the claim.
• The doc also gains the LRU-cliff table and the measured footprints, so the next person to reach for an LRU here finds the reason not to.
• diff's remaining peak is dominated by the loader cache, which is O(tree) by design. Reducing it further means not walking both roots in full — an algorithm change, not a cache change, and out of scope here.

Verification

• env GOCACHE=/tmp/cloudstic-gocache go test -race -count=1 ./... passes (full module)
• env GOCACHE=/tmp/cloudstic-gocache GOLANGCI_LINT_CACHE=/tmp/cloudstic-golangci-lint golangci-lint run ./... — 0 issues
• npx markdownlint-cli2 '**/*.md' — 0 issues

Docker-backed e2e (MinIO, SFTP) skipped locally for want of /var/run/docker.sock.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!