Context
PackStore.prepareFlushLocked (internal/storelayer/pack.go) writes every
bundled object into two maps:
for key, entry := range s.packKeys {
entry.PackRef = packRef
s.catalog[key] = entry
s.pendingShard[key] = entry
}
pendingShard exists to record which entries have not yet been written to a
shard. It pays for that by holding a second full copy of every entry the run
produced — keys, values and pack refs.
In a heap profile of backup over a 50,000-file tree this is the single largest
allocation in the process: prepareFlushLocked accounts for 51.9 MB flat out of
111 MB live. writeShard then marshals the map into a ~19 MB JSON blob and
seals that into another. Backup's peak RSS at that size is 360 MB against 161 MB
with packfiles disabled.
Related: #436 covers the same duplication on the read path.
Goal
A backup records which catalog entries are unflushed without holding a second
copy of them.
Scope
- Replace
pendingShard map[string]PackEntry with a record of which catalog
keys are unflushed (a key set, or a marker on the entry), reading values back
from s.catalog at flush time.
- Update
writeShard callers to build the shard from that set.
- Keep
discardPack correct: it currently removes the failed pack's entries from
both maps, and must still forget unflushed entries for a pack whose upload
failed.
Acceptance Criteria
- Entry values are stored once per run, not twice.
- Shard contents are unchanged for a given sequence of writes.
discardPack still forgets every entry pointing at an unwritten pack, so a
failed upload cannot leave the catalog asserting objects are stored.
go test ./internal/storelayer passes, including the existing pack durability
and upload-failure tests.
golangci-lint run ./internal/storelayer/... passes
Context
PackStore.prepareFlushLocked(internal/storelayer/pack.go) writes everybundled object into two maps:
pendingShardexists to record which entries have not yet been written to ashard. It pays for that by holding a second full copy of every entry the run
produced — keys, values and pack refs.
In a heap profile of
backupover a 50,000-file tree this is the single largestallocation in the process:
prepareFlushLockedaccounts for 51.9 MB flat out of111 MB live.
writeShardthen marshals the map into a ~19 MB JSON blob andseals that into another. Backup's peak RSS at that size is 360 MB against 161 MB
with packfiles disabled.
Related: #436 covers the same duplication on the read path.
Goal
A backup records which catalog entries are unflushed without holding a second
copy of them.
Scope
pendingShard map[string]PackEntrywith a record of which catalogkeys are unflushed (a key set, or a marker on the entry), reading values back
from
s.catalogat flush time.writeShardcallers to build the shard from that set.discardPackcorrect: it currently removes the failed pack's entries fromboth maps, and must still forget unflushed entries for a pack whose upload
failed.
Acceptance Criteria
discardPackstill forgets every entry pointing at an unwritten pack, so afailed upload cannot leave the catalog asserting objects are stored.
go test ./internal/storelayerpasses, including the existing pack durabilityand upload-failure tests.
golangci-lint run ./internal/storelayer/...passes