A field incident on 2026-07-19 turned out to be three defects stacked on top of each other. A full backup was flagged with one corrupt frame affecting one file. Re-checked afterwards, all four shards verify perfectly clean — 168/168 frames on both the recorded SHA-256 and a full zstd decode. The flag was a false positive from a transient read, and everything downstream of it went wrong.
1. A corrupt full stopped being a cycle — and could be deleted as collateral
cycles() opened a new cycle only for a full set that is_complete, which forbids any corrupt frame. So the flagged full was reparented onto the previous cycle as though it were an incremental:
- it did not appear in the cycle list at all;
- the previous cycle still reported "complete" while concealing it;
- every incremental taken afterwards was shown under a full it was not computed against — the restore chain was misrepresented;
- and because
Cycle.archivesincludesincr_sets, pruning the previous cycle would have deleted the later full with it. Withmax_cyclesand four complete cycles present, that was one backup away from happening.
Grouping now keys on a new ShardSet.is_restorable — "every shard was written and can be read" — rather than is_complete — "...and nothing was found corrupt." The cycle it opens still reports is_complete == False, so retention keeps protecting it and it is never treated as a clean base. failed/empty/in-progress fulls still do not open a cycle: those are untrustworthy, not merely damaged.
2. Verify condemned a frame on a single read
verify_frame_checksums() read each frame exactly once, immediately after writing ~22 GB across four concurrent shard threads, with the page cache dropped so the read crosses the network. One transient bad read became a permanent corrupt record.
A suspect frame is now re-read — evicting just that range first — and classified:
| Result | Verdict |
|---|---|
| A re-read matches the digest | read-recovered — stored bytes are fine, nothing condemned |
| Every read mismatches, identical bytes | Genuinely corrupt |
| Every read mismatches, different bytes | unstable-read — the read path is unreliable, so what is stored is unproven |
Both unconfirmed classes are deliberately biased away from "corrupt" and reported loudly instead: a false flag demotes a healthy full out of its own cycle and does not self-clear, which is worse than an anomaly warning.
verify_frame_checksums() now returns a 4-tuple (algo, count, bad, transient). Pass retries=0 for the old single-read behaviour.
3. A false positive could never be cleared
action_verify only printed — nothing ever wrote the manifest, so a bad flag was permanent short of hand-editing manifest.json, and while it was set the full stayed demoted.
--verify now reconciles: it clears a stale corrupt flag on a clean result, and marks a confirmed-corrupt archive that the manifest calls fine. It writes only on genuine disagreement, so a plain clean verify still touches nothing — and a system-class plan escalates only in that case, never for a no-op. The prior state is recorded in the entry's notes rather than silently overwritten.
Caveat: an entry marked corrupt loses whether it was ok or ok-with-warnings beforehand; it is restored as ok, with the prior state noted.
Verified end to end
On the real archives: all four shards OK, manifest reconciled, the cycle restored with its own incrementals, audit trail intact —
verify: manifest updated —
2026-07-19_full.s3of4.pax.zst: corrupt -> ok (corrupt, 1 corrupt frame(s))
Also: the README install example had been telling people to install v1.4.3.
388 tests pass (13 new).