Skip to content

Track unflushed catalog entries without duplicating them in pendingShard #437

Description

@rmanibus

Context

PackStore.prepareFlushLocked (internal/storelayer/pack.go) writes every
bundled object into two maps:

for key, entry := range s.packKeys {
    entry.PackRef = packRef
    s.catalog[key] = entry
    s.pendingShard[key] = entry
}

pendingShard exists to record which entries have not yet been written to a
shard. It pays for that by holding a second full copy of every entry the run
produced — keys, values and pack refs.

In a heap profile of backup over a 50,000-file tree this is the single largest
allocation in the process: prepareFlushLocked accounts for 51.9 MB flat out of
111 MB live. writeShard then marshals the map into a ~19 MB JSON blob and
seals that into another. Backup's peak RSS at that size is 360 MB against 161 MB
with packfiles disabled.

Related: #436 covers the same duplication on the read path.

Goal

A backup records which catalog entries are unflushed without holding a second
copy of them.

Scope

  • Replace pendingShard map[string]PackEntry with a record of which catalog
    keys are unflushed (a key set, or a marker on the entry), reading values back
    from s.catalog at flush time.
  • Update writeShard callers to build the shard from that set.
  • Keep discardPack correct: it currently removes the failed pack's entries from
    both maps, and must still forget unflushed entries for a pack whose upload
    failed.

Acceptance Criteria

  • Entry values are stored once per run, not twice.
  • Shard contents are unchanged for a given sequence of writes.
  • discardPack still forgets every entry pointing at an unwritten pack, so a
    failed upload cannot leave the catalog asserting objects are stored.
  • go test ./internal/storelayer passes, including the existing pack durability
    and upload-failure tests.
  • golangci-lint run ./internal/storelayer/... passes

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/storeperfPerformance, memory, and scaling work

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions