fix(storage): fail closed when the spend journal cannot be trusted

[Destynova2]

Closes #485.

ADR-0030 classifies the budget as an authorizing dependency: if it cannot be evaluated, the request must be blocked. The ADR's own audit note said the implementation already satisfied this "on every row", because the spend counter is in-memory and has no Redis to fail open on.

That in-memory counter is seeded by replaying spend/YYYY-MM.jsonl at startup, and the replay swallowed every error:

let file = match File::open(path) { Ok(f) => f, Err(_) => return SpendData::default() };
...
let Ok(event) = serde_json::from_str::<SpendEvent>(&line) else { continue };

So a truncated journal (crash mid-write), a corrupt line, or a permissions problem made grob start at $0 spent and silently reopen an already-exhausted budget. Same failure mode LiteLLM shipped, reached through the file layer instead of Redis, and invisible: no error, just a counter that reads zero.

Change

replay_current, replay_for_tenant and replay_all_tenants now return Result. The distinction that matters is absent vs untrustworthy:

• NotFound stays Ok(default) — a fresh install has genuinely spent nothing.
• Any other I/O error, or a line that will not parse is an error.

GrobStore::open propagates it, so a damaged journal refuses startup instead of serving on bad numbers. On the per-tenant path, which can hit the journal at request time on a cache miss, check_tenant_budget maps the failure to a BudgetError and denies.

load_spend was split: the global read is a pure cache hit and stays infallible, so only the read that can actually fail carries a Result.

Verification

src/storage/fault_injection_tests.rs (6 tests) injects each failure. Checked against origin/main in a scratch worktree: 2 fail there (corrupt_spend_journal_blocks_startup, corrupt_journal_denies_tenant_spend_read), confirming they catch the real bug and are not tautological. The other 4 pass on both sides, pinning the boundaries so the fix does not overshoot: a missing journal must still start clean, spend must survive a restart, and observing dependencies must never block a read.

The pre-existing malformed_lines_skipped test asserted the fail-open ("skip the broken line, carry on") and is deliberately inverted to malformed_line_is_an_error.

Full suite: 1733 passed, clippy -D warnings clean, 24 doc tests pass. ADR-0030's audit note and consequences updated to record the gap and its closure.

[github-actions]

Mutation testing (PR diff sample)

Informational — never blocks merge. Full matrix runs on main.

Metric	Value
Status	unknown
Duration	n/a s
Total	n/a
Caught	n/a
Missed	n/a
Timeout	n/a
Unviable	n/a

Legend: clean (no survivors), missed (inspect artifact), timed-out (25 min cap reached).

Artifact: mutants-pr-results-00b5d8d2eb8f9ed97384b1829b1da1cc8bf11e10.