Skip to content

Bound the filemeta loader cache the way the HAMT node cache is bounded #429

Description

@rmanibus

Context

Two caches in the read path hold decoded, content-addressed objects and need no
invalidation, because a ref's bytes can never change:

  • internal/hamt/nodestore.go caches node/ objects in an LRU capped at 4096
    (nodeCacheSize).
  • internal/engine/metaloader.go caches filemeta/ objects in a plain map that
    grows without limit.

Same problem, opposite policies. metaLoader is the expensive one: measured at
~456 bytes per entry for a minimal core.FileMeta (216 B shallow, plus Paths,
Parents, Extra and Xattrs in practice), against ~136 bytes for a bare key
set. backup, diff and prune construct the caching variant
(newMetaLoader) and touch every distinct filemeta across several snapshots, so
the map grows with the size of the repository rather than with anything bounded.

An LRU is safe here for the same reason it is safe in NodeStore: content
addressing means an eviction costs a re-read and can never yield a wrong answer.

See docs/caching.md for why this cache is not redundant with the pack LRU or
the node cache.

Goal

metaLoader's cache is bounded, with a size chosen from measurement rather than
picked, and the two decoded-object caches follow one policy.

Scope

  • Add a benchmark covering the access pattern that motivates the cache: diff
    and prune across several consecutive snapshots, where an unchanged file
    keeps its filemeta from one snapshot to the next.
  • Record the hit rate and allocation profile at the current unbounded size as
    the baseline.
  • Replace the map with an LRU (hashicorp/golang-lru/v2, already a dependency
    via internal/hamt), sized from the benchmark.
  • Keep newUncachedMetaLoader as-is — ls, restore and find deliberately
    read through, for reasons documented on the type.
  • Update docs/caching.md's inventory table with the new bound.

Acceptance Criteria

  • metaLoader's memory is bounded by a constant, not by repository size
  • the chosen size is justified by benchmark output quoted in the PR
  • benchstat shows no wall-time regression on the diff and prune benchmarks
  • go test -race ./internal/engine passes
  • golangci-lint run ./internal/engine/... passes

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/coreCore backup engine, repository model, and restore semanticsenhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions