Relay bricks permanently when its audit chain is corrupted by an unclean stop; no supported recovery path

[jeremi]

Observed

In a long-running compose lab (Solmara), two Relay containers came back from an unclean stop with a torn last line in their audit.jsonl. From that point every audited request failed closed:

audit.write_failed: audit chain verification failed: line 103 prev_hash mismatch

and callers saw 5xx (audit.write_failed from the relay; upstream Notary evaluations surfaced it as source.unavailable). The services stayed in that state indefinitely; health checks reported the containers healthy while every real request failed.

Why (code path on main)

JsonlFileSink recomputes the retained chain tail before appending (tail_hash_from_files / verify_chain in crates/registry-platform-audit/src/lib.rs). Any corruption in the retained JSONL set makes every subsequent append return AuditError::ChainVerification, which the relay maps to audit.write_failed (crates/registry-relay/src/audit/mod.rs). Size-based rotation exists, but there is no path that ever moves a corrupt segment aside.

What's missing

Failing closed is the right default for audit integrity; the gap is operability:

1. No supported recovery. The only way out today is manual surgery inside the audit volume (stop the relay, move audit.jsonl aside, restart). Nothing documents this, and doing it silently discards the operator-visible record of the discontinuity. A supported operation (CLI or admin endpoint) should quarantine the corrupt segment, record an explicit chain-break event (why, when, tail hash before/after), and restart the chain, ideally anchored via the existing ChainVerificationAnchors mechanism.
2. No startup detection. The relay only discovers the broken chain on the first audited request. Verifying the retained chain at startup (and reflecting it in readiness/health) would turn a confusing per-request 5xx into an actionable operational signal.
3. Crash consistency. The corruption source was an unclean container stop, which is routine in container platforms. The append path could be hardened against torn writes (e.g. tolerate and quarantine a partial final line as a distinct, explicitly-logged case rather than a generic prev_hash mismatch).

Notes

Audit integrity is a security-sensitive area per CONTRIBUTING; any fix needs explicit review notes on how the chain break is itself made tamper-evident.

[jeremi]

Second trigger observed for the same permanent-brick outcome, this time without any torn write: the relays were briefly recreated with an empty environment (a compose invocation that bypassed the generated .env), so the audit HMAC secret was empty during a crash-loop window. Lines appended in that window chain under the wrong key, and once the correct secret came back every append fails with a mid-file prev_hash mismatch (line 182, not the final line). Same dead end: healthy healthcheck, every audited request 5xx, manual volume surgery to recover.

So besides crash consistency, a keyed-hasher configuration change (secret rotation, secret temporarily missing) also produces an unrecoverable chain with no operator path. The recovery mechanism in this issue should treat "retained chain does not verify under the currently configured key" as the general case, not just torn tails.

[jeremi]

Correction to my previous comment: the empty-HMAC-secret mechanism I described does not hold. The platform audit crate fails closed at startup on empty or weak hash secrets (EmptySecret / WeakSecret in registry-platform-audit), so nothing can append under a missing key. Retained-file timestamps also show the mismatched lines were written well before that incident window; the restart merely exposed a pre-existing inconsistency at verification time.

So both triggers reported here reduce to the same shape: an inconsistency gets retained while the process is running, and the next restart turns it into a permanent brick. The root cause of the retained inconsistency is being tracked separately; the asks in this issue stand unchanged and are needed regardless of trigger (startup detection with an operator-visible path, a supported quarantine/recovery operation, and treating "retained chain does not verify under the configured key" as the general recovery case).

[jeremi]

Triage note: #196 and #211 stay separate issues by design. #211 is prevention and detection (single-writer enforcement, write-time chain checks); this issue is the operator recovery path for a chain that is already inconsistent. A deployment on an affected version needs the recovery path even after prevention ships, so the two have independent acceptance criteria. Cross-linked; same workstream.