RFC 0018: Epic / Tracking issue for self-describing packfiles and a rebuildable pack index

[rmanibus]

Context

Phase 1 of #287 removes the paths that silently destroy data, but leaves the underlying design intact: index/packs is a single mutable JSON blob that is the sole record of every packed objects location. Lose it and the bytes are unrecoverable, because packfiles (pkg/store/pack.go) carry no framing, no magic marker, and no footer.

This is the one index in the repository with no self-healing path. index/snapshots reconciles against LIST snapshot/ (internal/engine/snapshots.go); pack entries cannot, because offsets exist nowhere else. It also makes the concurrent-backup race unfixable in place: backups take shared locks by design (internal/engine/repolock.go:126), and the catalog is read-modify-write with last-writer-wins, so the losing runs entries — including its own snapshot — become unaddressable.

Goal

Make index/packs a rebuildable cache rather than a single point of failure, and remove the read-modify-write race on it.

Scope

• Write rfcs/0018-self-describing-packfiles.md and list it in rfcs/README.md
• Append a footer to each packfile: a JSON entry list of (key, offset, length), the footers own length as trailing fixed-width bytes, and a magic marker
• Define the recovery path: on catalog mismatch or loss, LIST packs/ and read footers to rebuild
• Shard the catalog: each run writes its own immutable index/packs/<pack-hash> object once at flush time; readers merge shards; prune compacts them under its exclusive lock. Append-only shards remove the race without needing a compare-and-swap the ObjectStore interface cannot assume
• Define the migration: new packs get footers, existing footerless packs stay readable via the legacy catalog until a repack migrates them
• Specify how check uses footers to verify pack integrity, and sketch a future repair-index command
• Revisit backup flush ordering as part of this: flush HAMT, flush packs and write the shard, write the snapshot, update index/latest

Acceptance Criteria

• RFC 0018 merged, following the rfcs/ conventions in AGENTS.md
• Implementation issues filed against the RFC and linked here
• A two-process concurrent-backup e2e test asserts both snapshots are fully restorable
• A test asserts the catalog can be rebuilt from packfile footers after index/packs is deleted
• go test ./pkg/store ./internal/engine ./e2e passes
• golangci-lint run ./pkg/store/... ./internal/engine/... passes

Part of #287

[rmanibus]

Implementation breakdown

Filing these against the RFC as the acceptance criteria call for. Ordered by dependency:

• Add self-describing packfile footers and rebuild-from-footers recovery (RFC 0018) #298 — Self-describing footers and rebuild-from-footers recovery (stages 1 and 3). In review as feat(store): make packfiles self-describing so the catalog can heal (RFC 0018) #297.
• Implement RangeGetter for the S3, B2, and SFTP backends #299 — RangeGetter for S3, B2, and SFTP. Independent of the format; makes footer recovery cheap on the backends where it matters.
• Shard the pack catalog into append-only per-pack objects (RFC 0018) #300 — Sharded, append-only catalog (stage 2). Depends on Add self-describing packfile footers and rebuild-from-footers recovery (RFC 0018) #298 for the survivability guarantee underneath it.
• Add doctor command and pack integrity verification to check #301 — repair-index and pack verification in check. Depends on Add self-describing packfile footers and rebuild-from-footers recovery (RFC 0018) #298 for RebuildCatalog.

Stage 3 of the RFC — retiring the monolithic index/packs behind an explicit repository upgrade — is deliberately not filed yet. It is the only breaking step, and it should wait until #300 has settled and enough repositories have migrated their footerless packs.

Two open questions from the RFC are still unresolved and block nothing so far:

• Footer encryption — follows index/packs is written unencrypted in encrypted repositories #295. The footer carries the same object keys and per-object sizes that index/packs already exposes in plaintext today, so whatever is decided there applies to both.
• Footer encoding — JSON today. A binary encoding would be meaningfully smaller for packs holding thousands of entries, at the cost of hand-debuggability.