You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The memory benchmark shows check's cumulative allocation jumping an order of
magnitude between two adjacent size points, for 2.5x the data:
files
alloc
5,000
117.9 MB
20,000
270.9 MB
50,000
2,949 / 2,984 / 2,924 MB
prune shows the same jump and is bimodal at 50,000 — 3,204 MB on one sample,
881 MB on the other two.
This is not a benchmark artifact. Reproduced locally by building the repository
the benchmark builds — an initial backup followed by an incremental that rewrites
every 20th file, which is what puts the object count over the threshold:
The same tree with a single snapshot produces 4 packs and allocates 549 MB. Same
object count, ~9x the allocation, and the only difference is that the pack count
crossed the size of the body cache.
go tool pprof -alloc_space attributes it unambiguously:
4.24GB 87.80% os.readFileContents
reached through PackStore.Get → resolveFromPack (4.22 GB cumulative). At
roughly 9 MB per packfile that is about 470 whole-pack reads for a repository
containing 7 packs.
Cause
PackStore.packCache holds a fixed four packfile bodies. Once a repository
has more packs than that and an operation's access order moves between them, the
cache thrashes, and a miss re-reads an entire packfile to return one small
object.
Ranged reads (#452) do not prevent this. They serve the first misses on a pack,
but promotion to a whole-pack fetch fires after packPromoteAfter misses — and
under thrashing the promoted pack is evicted and re-promoted repeatedly, so each
round costs another full transfer.
The threshold is low: four packs is roughly 32 MB of packed small objects, so any
non-trivial repository is past it. The benchmark's 50,000-file point is simply
where the curve crosses it, which is why the jump looks superlinear rather than
gradual.
On a local store this is page-cached reads and shows up as allocation. On S3 or
B2 it is 9 MB of egress per miss, billed.
Goal
check, and any operation whose working set spans more packs than the body cache
holds, does not re-transfer whole packfiles to read small objects.
Scope
Three directions, not mutually exclusive; the issue is which combination is right.
Fix the access order.perf(engine): restore leaves in walk order, not tree order #455 did this for restore: ordering reads to follow
pack layout took its write-phase miss rate from 55.5% to 0.74%. check walks
HAMT nodes, then per leaf a filemeta, its content object and its chunks — worth
tracing to see whether that interleaves packs, and whether it can be grouped.
Reconsider promotion under eviction pressure. Promoting to a whole-pack
fetch is right for a pack being scanned and wrong for one being thrashed; PackStore currently cannot tell the difference. Tracking evictions per pack
would.
Size the body cache against the working set. Note that shrinking it is
measurably worse (#440 comment),
and that growing it trades resident memory for transfers, so this needs a
number rather than a guess.
Measure with the trace method described in that comment — record which pack each
resolution lands in and simulate cache sizes offline. -debug line counts are
not reliable for this.
Acceptance Criteria
check on a repository with substantially more packs than the body cache does
not re-read whole packfiles proportional to its object count; total allocation
grows with repository size rather than jumping when pack count crosses the
cache size.
The benchmark's check and prune allocation figures at 50,000 files are in
line with the 20,000-file figures scaled by size.
prune's bimodality at 50,000 files is gone, or explained.
go test ./internal/storelayer ./internal/engine passes
golangci-lint run ./internal/storelayer/... ./internal/engine/... passes
Context
The memory benchmark shows
check's cumulative allocation jumping an order ofmagnitude between two adjacent size points, for 2.5x the data:
pruneshows the same jump and is bimodal at 50,000 — 3,204 MB on one sample,881 MB on the other two.
This is not a benchmark artifact. Reproduced locally by building the repository
the benchmark builds — an initial backup followed by an incremental that rewrites
every 20th file, which is what puts the object count over the threshold:
The same tree with a single snapshot produces 4 packs and allocates 549 MB. Same
object count, ~9x the allocation, and the only difference is that the pack count
crossed the size of the body cache.
go tool pprof -alloc_spaceattributes it unambiguously:reached through
PackStore.Get→resolveFromPack(4.22 GB cumulative). Atroughly 9 MB per packfile that is about 470 whole-pack reads for a repository
containing 7 packs.
Cause
PackStore.packCacheholds a fixed four packfile bodies. Once a repositoryhas more packs than that and an operation's access order moves between them, the
cache thrashes, and a miss re-reads an entire packfile to return one small
object.
Ranged reads (#452) do not prevent this. They serve the first misses on a pack,
but promotion to a whole-pack fetch fires after
packPromoteAftermisses — andunder thrashing the promoted pack is evicted and re-promoted repeatedly, so each
round costs another full transfer.
The threshold is low: four packs is roughly 32 MB of packed small objects, so any
non-trivial repository is past it. The benchmark's 50,000-file point is simply
where the curve crosses it, which is why the jump looks superlinear rather than
gradual.
On a local store this is page-cached reads and shows up as allocation. On S3 or
B2 it is 9 MB of egress per miss, billed.
Goal
check, and any operation whose working set spans more packs than the body cacheholds, does not re-transfer whole packfiles to read small objects.
Scope
Three directions, not mutually exclusive; the issue is which combination is right.
pack layout took its write-phase miss rate from 55.5% to 0.74%.
checkwalksHAMT nodes, then per leaf a filemeta, its content object and its chunks — worth
tracing to see whether that interleaves packs, and whether it can be grouped.
fetch is right for a pack being scanned and wrong for one being thrashed;
PackStorecurrently cannot tell the difference. Tracking evictions per packwould.
measurably worse (#440 comment),
and that growing it trades resident memory for transfers, so this needs a
number rather than a guess.
Measure with the trace method described in that comment — record which pack each
resolution lands in and simulate cache sizes offline.
-debugline counts arenot reliable for this.
Acceptance Criteria
checkon a repository with substantially more packs than the body cache doesnot re-read whole packfiles proportional to its object count; total allocation
grows with repository size rather than jumping when pack count crosses the
cache size.
checkandpruneallocation figures at 50,000 files are inline with the 20,000-file figures scaled by size.
prune's bimodality at 50,000 files is gone, or explained.go test ./internal/storelayer ./internal/enginepassesgolangci-lint run ./internal/storelayer/... ./internal/engine/...passes