Skip to content

fix(backup): no backup check is a failure by default - #479

Merged
passcod merged 1 commit into
mainfrom
fix/backup-checks-warn-by-default
Aug 4, 2026
Merged

fix(backup): no backup check is a failure by default#479
passcod merged 1 commit into
mainfrom
fix/backup-checks-warn-by-default

Conversation

@passcod

@passcod passcod commented Aug 4, 2026

Copy link
Copy Markdown
Member

🤖 Follow-up to #478.

Why

A failure in canopy means a live service is down, and tech support acts on it within minutes. Backup signals were shipping as failures, so a six-hourly backup slipping one cycle — on a fleet with layered backups, where the next run fixes it — sent someone looking for an outage that was never happening.

The rule

No backup signal is a failure by default. Every check in the backup sphere registers at a warning ceiling and does not escalate:

before after
backup-staleness, backup-maintenance-stale, backup-maintenance-error, backup-reconcile-missing, preflight-assume, restore-verification Failed Warning
preflight-identity Failed + escalates Warning
backup-corruption, backup-rotation-broken, preflight-object-lock Failed + escalates unchanged
backup-never, report-gap, size-mismatch, redaction, migration-test Warning unchanged

The three exceptions default to an escalating failure because the backups are already gone, unrecoverable, or unprotected, rather than merely late.

Only the ceiling moves. The sweeps still observe a failure, so an operator raising the ceiling through policy gets the failure back with no code change — that is their call to make with the people who answer the alerts, not a default to ship.

The blip case comes free

Warnings never open an incident on their own (opens_incident() requires Failed), and escalates is already inert below Failed (escalates_normalised). So this delivers "don't page on a blip" without any new grace mechanism: the signal stays visible on the server's health rollup and in the check listing, and only reaches an incident if the group already has one open for something else.

Written down so it survives

  • the alerting section of the BKJ spec (normative prose);
  • the backup/refs.rs module doc, next to the constants;
  • AGENTS.md, alongside a second new rule forbidding parameterised check names — the fifteen-alerts-for-fifteen-configurations problem, which is the next change;
  • a test that pins the rule against what the sweeps actually seed into the catalog, rather than restating the defaults, so a new check cannot quietly ship as a failure.

Migration

CheckPolicy::register only seeds a catalog row on first sight, so changing the shipped defaults in code does nothing to rows that already exist. The migration resets them — but only where reviewed_by is still canopy. register() stamps that on insert, so any other value means an operator set the policy deliberately, and raising a backup check to a failure in consultation with tech support is exactly the call they are entitled to make. Those rows are untouched.

It also re-grades already-open failed issues down to warning and closes the incidents that were only being held open by them, so existing noise clears on deploy instead of waiting for each check to next file. No Slack resolve is enqueued — those incidents should not have been opened.

Verified against a populated catalog in a rolled-back transaction: canopy-defaulted rows reset, the three exceptions untouched, an operator-reviewed row left exactly as set, and non-backup checks like reachability unaffected.

Test changes

Five existing assertions moved from failed to warning. Rather than flipping the string, they now assert the full contract: the observation is still failed, the effective result is warning, and no incident opens. The issue helper reads observed_result and escalates alongside what it already read.

A failure in canopy means a live service is down, and tech support acts
on it within minutes. Backup signals were shipping as failures, so a
six-hourly backup slipping one cycle — on a fleet with layered backups,
where the next run fixes it — sent someone looking for an outage that
was never happening.

Every check in the backup sphere now registers at a warning ceiling and
does not escalate: staleness, never-backed-up, both maintenance checks,
reconcile-missing, both preflight checks other than object-lock, and
restore-verification. The observation is unchanged — the sweeps still
observe a failure — so raising the ceiling through policy gets the
failure back with no code change. That is the operator's call to make
with the people who answer the alerts, not a default to ship.

Three keep defaulting to an escalating failure, because the backups are
already gone, unrecoverable, or unprotected rather than merely late:
backup-corruption, backup-rotation-broken, preflight-object-lock.

Warnings never open an incident on their own, so this also delivers the
'do not page on a blip' half without any new grace mechanism: the signal
stays visible on the server's health and in the check listing, and only
reaches an incident if the group already has one open.

The rule is written down in three places so it survives: the alerting
section of the BKJ spec, the refs module doc, and AGENTS.md. A test
pins it against what the sweeps actually seed into the catalog, so a
new check cannot quietly ship as a failure.

CheckPolicy::register only seeds on first sight, so the migration resets
the catalog rows that already exist — but only where reviewed_by is
still 'canopy'. A row an operator reviewed is their deliberate choice
and is left alone. It also re-grades already-open failed issues down and
closes the incidents that were only being held open by them, so the
existing noise clears on deploy instead of waiting for each check to
re-file.
@passcod
passcod added this pull request to the merge queue Aug 4, 2026
Merged via the queue into main with commit 2fa510d Aug 4, 2026
7 checks passed
@passcod
passcod deleted the fix/backup-checks-warn-by-default branch August 4, 2026 03:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant