fix(storage): size the storage expectation from measured corpus

[rickyrombo]

The gap

Storage expectation reads ~16% below what nodes actually hold. Across all 35 file-backed mainnet nodes, disk-used ÷ expectation runs 1.07–1.34×, and not one is below 1.0.

node	expectation	disk used	ratio
audius-content-11.figment.io	2.152 TB	2.892 TB	1.34×
audius-content-8.figment.io	2.152 TB	2.691 TB	1.25×
…33 more	2.152 TB	2.30–2.68 TB	1.07–1.24×

The disk was never the mystery. On every node with Postgres on the blob volume, mediorumPathUsed reconciles to within 0.3% of lastSuccessfulRepair.ContentSize + databaseSize — repair's own tally of blobs it decided to keep. e.g. content-8: 2,689,227,333,632 used vs 2,570,710,940,019 + 125,780,483,431 = 2,696,491,423,450. The expectation was modelling the wrong corpus.

What was wrong

(SUM(ff_probe.size) * 2 * replication_factor) / node_count

The * 2 stood in for "we store a transcode as well as the original", and was wrong in both directions at once:

• Over-counted transcodes. Most originals are lossless or high-bitrate and downsample well below their own size — measured across mainnet, transcodes are 0.66 TB against 1.06 TB of originals, a ratio of 0.63, not 1.0. It also doubled image uploads, which have no transcode at all.
• Omitted more than it over-counted. Repair iterates uploads, audio_previews and qm_cids (repair.go:331, :379, :442). Only the first was in the numerator.

What it does now

  originals                   exact, from ffprobe
+ audio_duration * 40_000     320 kbps CBR (transcode.go "-b:a 320k")
+ previews * 1_200_000        30 s at that bitrate
+ legacy Qm corpus
  all * replication_factor / node_count

Each constant is the consequence of a decision made elsewhere in the package, not a fitted parameter.

The legacy corpus

The term that matters most, and the hardest to get. Those blobs are tracked only as keys in qm_cids, with no size recorded anywhere in Postgres — invisible to any query over uploads, despite being roughly a third of what a node stores.

Measured by sampling 20,000 of the 2,445,544 addressable keys against a storeAll node's /internal/blobs/info, at a 99.9% hit rate on both classes:

class	in list	sampled	hit%	mean	total
bare Qm CIDs (audio)	1,045,437	8,498	99.9%	14.654 MB	15.31 TB
…/original.jpg (artwork)	1,400,107	11,502	99.9%	0.685 MB	0.96 TB
resized variants	3,898,515	1,221	0.0%	—	0.00 TB

Variants are excluded because repair deletes them on sight — they're regenerated on demand (repair.go:708) — and 0 of 1,221 sampled existed anywhere.

It's derived from qm_cids row counts rather than hardcoded as a total, so dev and stage correctly get zero instead of inheriting all of mainnet's history. The counts are stable (qm_cids is filled once by drop_blobs.sql and never appended to), so it's computed once and kept.

Result

2.60 TB/node against content-8's measured 2.571 TB — +1.2%, versus −16.3% today. It also sits inside the 2.278–2.628 TB range measured across all file-backed nodes.

Independent cross-check: this puts legacy at 33.0% of the corpus. A filesystem walk of a different node's blob tree measured legacy at 1052.5 GB of 3079.1 GB = 34.2%. Two unrelated instruments — HTTP sampling on one operator's node, du on another's — agreeing to 1.2 percentage points.

Scope

Still a fair-share model. It describes a node holding its rendezvous share and says nothing about StoreAll, StoreRecent or archive tiers, whose footprint is set by retention config. For a StoreAll node the right shape is corpus × rf/n for primary and corpus × (1 − rf/n) for archive — worth a follow-up, and it needs the per-bucket content split that ContentSize currently conflates.

The legacy constant can only shrink (prune and declared data loss remove blobs; nothing adds them), so it will drift slowly and is worth re-measuring periodically.

Tests

computeStorageExpectation is split from the SQL so the arithmetic is testable without a database: rendezvous share, degenerate inputs (zero nodes / zero rf / empty network), each component counted exactly once, images not doubled, and a case pinning the real mainnet numbers to within 5% of measured while asserting the old formula is demonstrably >10% low.

go build, go vet and gofmt clean. The package has three failures — TestPollDelistStatuses, TestRepair, TestUploadFile — which reproduce identically on main; they need the Docker harness (make test-mediorum). No new failures.

🤖 Generated with Claude Code