fix: link the GC chain on full rebuild (a rebuild orphans the ledger's whole index history)

[christophediprima]

Summary

fluree-db-indexer/src/build/rebuild.rs publishes every full-rebuild root with no prev_index:

None, // GC chain deferred for V3 milestone.

A root with no prev_index starts a fresh lineage. Every index version before it becomes unreachable in one stroke — referenced by no root and listed in no garbage manifest. Since nothing in the indexer reclaims by reachability, those artifacts can never be freed by any retention setting, at any value. It is not a slow leak: one rebuild discards the ledger's entire index history, permanently.

Why this is worse than "rebuilds are rare"

Two amplifiers make it self-feeding rather than occasional:

• A recoverable memory error converts into permanent disk loss. ENOMEM during an incremental index falls back to a full rebuild. So a transient allocation failure — one that would otherwise be retried harmlessly — permanently strands the ledger's history instead.
• A wedged volume forces rebuilds. The fuller the disk, the more rebuilds, the more orphans, the fuller the disk. Memory pressure causes disk pressure causes more rebuilds.

The end state is unrecoverable without external intervention: GC must write in order to free anything, so once the volume is full, every index attempt fails, GC never runs, and it does not recover on its own. Ours sat dead for ~34 hours while reporting healthy.

Measured, before and after, on the same live ledger

Reconstructed by decoding every root blob's index_t and prev_index out of live storage (examples/root_graph.rs, added in this PR) on a ledger under continuous ingest.

Before, on the unpatched build:

roots on disk    : 226
undecodable      : 0
roots with no prev_index (expect 1 = genesis): 2      <- TWO lineages, not one
fork points (parent with >1 child): 0                 <- NOT competing builds
dangling prev pointers (chain stops here): 7
reachable from head 4acaf0f11fc48d81 : 15 of 226 roots  (211 UNREACHABLE)
distinct unreachable index_t: 211 (so 0 share a t with another root)

93% of index versions unreachable. The two lines that pin the diagnosis are fork points: 0 and 0 share a t with another root: those rule out the obvious competing-builds explanation (two builds from one parent, loser abandoned), which is the theory we spent a day on. A second root with no parent link at all is the signature of this bug specifically.

After, same ledger, same command, patched build:

roots on disk    : 21
undecodable      : 0
roots with no prev_index (expect 1 = genesis): 0
fork points (parent with >1 child): 0
dangling prev pointers (chain stops here): 1
reachable from head b59e2bfb8d4530f8 : 21 of 21 roots  (0 UNREACHABLE)

211 unreachable → 0.

Reading the two residuals, since neither is a defect: no prev_index: 0 rather than 1 is correct here — GC has legitimately truncated past genesis, so the oldest retained root points at a collected parent, which is the same fact as dangling: 1. One dangling pointer at the tail is what a bounded chain looks like; seven scattered through the middle is what a broken one looks like.

Being precise about what this fix accounts for: the 226 → 21 drop is this fix plus a separate sweep that reclaimed the already-stranded artifacts. The 0 UNREACHABLE is this fix alone — no sweep can make a root reachable.

What changed, and why the shape matters

The naive fix is to pass prev_root_id at the one call site. That is not sufficient, because of how the two concerns were coupled:

encode_and_write_root_v6 took gc_ctx: Option<GarbageContext>, and that single Option carried two unrelated things: the cheap prev_index link, and the expensive garbage diff computed by walking both roots' CAS id sets. Any failure in the diff — an unreadable prior root, a missing leaf — dropped the link too. So the chain could break silently even where a caller did pass a previous root.

This PR:

• Sets prev_index as soon as the previous root decodes, and computes the garbage diff best-effort on top of that. The link no longer depends on the diff succeeding.
• Changes the signature to prev_root_id: Option<ContentId>, deriving the garbage context internally. A caller can no longer pass a previous root and forget the link — the shape that produced this bug is no longer expressible.
• Un-dead-codes compute_garbage_from_prev_root, which existed but was unreachable.

Tests

test	pins
decodable_prev_head_always_yields_a_prev_index_link	if the previous root decodes, the link is set — independent of whether the garbage diff succeeded
unreadable_prev_head_yields_no_link_rather_than_a_bogus_one	an unresolvable previous head produces no link, not a dangling one

fluree-db-indexer: 341 passed, 2 ignored. fmt clean; clippy --all-features --all-targets clean on the changed crates.

Honest note on coverage: we could not construct a case where the garbage diff fails while the previous root decodes, because collect_root_cas_ids_expanded tolerates missing leaves. So the "diff failed but link still set" branch is correct by construction rather than by test. Called out rather than papered over with a test that asserts something weaker than its name suggests.

examples/root_graph.rs

Added because this bug was not diagnosable from logs or counts. GC reported chain_len=21 while the ledger held 226 root files, and no amount of reasoning about the truncation loop explained the gap — six hypotheses, all about deletion or concurrency, all wrong, because the bug was in creation.

Every root carries index_t and prev_index, so the real structure is recoverable. The tool decodes a roots directory and reports which of three shapes it is — one broken chain, multiple lineages, or unchained roots — because those have entirely different fixes.

Kept in the PR as a diagnostic, not as shipped behaviour. If you would rather it not live in the repo, we will drop it, but it is what turned a week of speculation into one measurement.

Notes for review

• The comment says "deferred for V3 milestone", so this may be a known and intentional gap rather than an oversight. If there is a reason full rebuilds should not chain — a format or migration constraint we have not found — we would rather hear it than have this merged. The evidence above is that in a running deployment it costs the ledger's whole history, but we do not know what V3 assumed.
• Consider warn! on a rebuild that cannot chain. Right now a severed chain is invisible until you decode the root graph. This PR does not add that, to keep the change minimal, but a deployment would have caught the problem in a day instead of a week.

[IX-Erich]

Heads up — #1614 touches the same lines and looks like a superset of this change.

We hit this independently and have been carrying the identical rebuild.rs fix in production since Aug 2, so this is a third-deployment confirmation of your diagnosis: 3,019 index roots against 2,811 garbage manifests — 208 roots stranded behind manifest-less ones — and 96 GB on disk for 201 MB of actual commit data. Once a ledger is reindexed, retention can never truncate past that root again. Your "one rebuild discards the ledger's entire index history, permanently" matches what we measured exactly.

Comparing the branches: this one passes prev_root_id.clone() where stock passed None, which links the chain. #1614 does the same (rebuild.rs:122 → 1134) and additionally writes a real garbage manifest diffed from the prior root's reachable set, removes the two conditions that stall the collector permanently (missing manifest, zero created_at_ms), and adds a reachability-based sweep — the only thing that can reclaim artifacts already orphaned by the old behaviour, which chain repair alone can't reach. Our patch, like this one, left the manifest empty and repaired traversal only.

I applied our six-patch stack against both branches: pr1614 conflicts only on this file, everything else clean. So we're planning to drop ours when #1614 merges.

One thing #1614 does not absorb: I grepped it for hard_max_old_indexes/hard_keep and found nothing, and the age guard survives in collector.rs ("Garbage record too recent, stopping GC"). So #1601 stays necessary on top of it — #1614 removes the missing-manifest and zero-timestamp stalls, but not the AND that lets the age guard override the count target. Worth landing the two together, since neither bounds disk alone.

[zonotope]

Thanks for the contribution @christophediprima, but I'm closing this pull request in favor of the recently merged #1614. The fixes on that branch are a superset of those here, except for the tool defined in root_graph.rs. I think those tools are a useful addition, so I've also opened #1622 to add it.