fix(backup): no backup check is a failure by default - #479
Merged
Conversation
A failure in canopy means a live service is down, and tech support acts on it within minutes. Backup signals were shipping as failures, so a six-hourly backup slipping one cycle — on a fleet with layered backups, where the next run fixes it — sent someone looking for an outage that was never happening. Every check in the backup sphere now registers at a warning ceiling and does not escalate: staleness, never-backed-up, both maintenance checks, reconcile-missing, both preflight checks other than object-lock, and restore-verification. The observation is unchanged — the sweeps still observe a failure — so raising the ceiling through policy gets the failure back with no code change. That is the operator's call to make with the people who answer the alerts, not a default to ship. Three keep defaulting to an escalating failure, because the backups are already gone, unrecoverable, or unprotected rather than merely late: backup-corruption, backup-rotation-broken, preflight-object-lock. Warnings never open an incident on their own, so this also delivers the 'do not page on a blip' half without any new grace mechanism: the signal stays visible on the server's health and in the check listing, and only reaches an incident if the group already has one open. The rule is written down in three places so it survives: the alerting section of the BKJ spec, the refs module doc, and AGENTS.md. A test pins it against what the sweeps actually seed into the catalog, so a new check cannot quietly ship as a failure. CheckPolicy::register only seeds on first sight, so the migration resets the catalog rows that already exist — but only where reviewed_by is still 'canopy'. A row an operator reviewed is their deliberate choice and is left alone. It also re-grades already-open failed issues down and closes the incidents that were only being held open by them, so the existing noise clears on deploy instead of waiting for each check to re-file.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🤖 Follow-up to #478.
Why
A failure in canopy means a live service is down, and tech support acts on it within minutes. Backup signals were shipping as failures, so a six-hourly backup slipping one cycle — on a fleet with layered backups, where the next run fixes it — sent someone looking for an outage that was never happening.
The rule
No backup signal is a failure by default. Every check in the backup sphere registers at a warning ceiling and does not escalate:
backup-staleness,backup-maintenance-stale,backup-maintenance-error,backup-reconcile-missing,preflight-assume,restore-verificationpreflight-identitybackup-corruption,backup-rotation-broken,preflight-object-lockbackup-never,report-gap,size-mismatch,redaction,migration-testThe three exceptions default to an escalating failure because the backups are already gone, unrecoverable, or unprotected, rather than merely late.
Only the ceiling moves. The sweeps still observe a failure, so an operator raising the ceiling through policy gets the failure back with no code change — that is their call to make with the people who answer the alerts, not a default to ship.
The blip case comes free
Warnings never open an incident on their own (
opens_incident()requiresFailed), andescalatesis already inert belowFailed(escalates_normalised). So this delivers "don't page on a blip" without any new grace mechanism: the signal stays visible on the server's health rollup and in the check listing, and only reaches an incident if the group already has one open for something else.Written down so it survives
BKJspec (normative prose);backup/refs.rsmodule doc, next to the constants;AGENTS.md, alongside a second new rule forbidding parameterised check names — the fifteen-alerts-for-fifteen-configurations problem, which is the next change;Migration
CheckPolicy::registeronly seeds a catalog row on first sight, so changing the shipped defaults in code does nothing to rows that already exist. The migration resets them — but only wherereviewed_byis stillcanopy.register()stamps that on insert, so any other value means an operator set the policy deliberately, and raising a backup check to a failure in consultation with tech support is exactly the call they are entitled to make. Those rows are untouched.It also re-grades already-open failed issues down to warning and closes the incidents that were only being held open by them, so existing noise clears on deploy instead of waiting for each check to next file. No Slack resolve is enqueued — those incidents should not have been opened.
Verified against a populated catalog in a rolled-back transaction: canopy-defaulted rows reset, the three exceptions untouched, an operator-reviewed row left exactly as set, and non-backup checks like reachability unaffected.
Test changes
Five existing assertions moved from
failedtowarning. Rather than flipping the string, they now assert the full contract: the observation is stillfailed, the effective result iswarning, and no incident opens. The issue helper readsobserved_resultandescalatesalongside what it already read.