💡 Canary Burn-Rate SLO with Post-Promotion Health Monitoring and Auto-Rollback

[github-actions[bot]]

Summary

Extend the canary-rollout scheduled auto-promote (#1056) with a post-promotion health monitoring gate: after a canary ring promotion, continuously track the promoted version's error budget (workflow failure rate, review quality regressions, token burn anomalies) against a configurable burn-rate SLO, and automatically rollback to the previous channel tag if the error budget is exhausted within the observation window. This complements the existing eval-score-gated promotion (#1042) — which prevents bad promotions — by catching regressions that emerge only under production load.

Market Signal

Progressive delivery with automated rollback is industry standard for traditional deployments (Argo Rollouts, Flagger), but applying it to AI agents that modify their own infrastructure is an emerging practice with unique failure modes:

• Antigravity Lab (2026): Published a canary deployment framework with burn-rate SLO auto-rollback specifically for AI agents, using error-budget windows rather than instantaneous thresholds to tolerate environmental noise
• AEROS paper (arXiv 2605.28097): Describes a governed-evolution pipeline for autonomous agents with validator → sandbox → shadow → rollback sequences, partitioning agent capabilities into independently deployable modules
• Key insight: Agent failures compound — a bad review agent approves a bad PR, which breaks CI, which creates more work for the agent fleet. Post-promotion monitoring must account for this cascading failure mode, not just individual workflow failures

Source: Antigravity — AI Agent Canary with Burn-Rate SLOs, AEROS — Identity-Stable Canary for Embodied Agents

User Signal

The canary-rollout infrastructure has matured rapidly through recent PRs:

• Auto-cut on reusable change (feat: implement issue #1069 — canary-rollout: auto-cut a new version when a reusable changes on main (close the last manual seam) #1070)
• Scheduled fleet auto-promote (feat(canary-rollout): gated scheduled fleet auto-promote (#1045 part b) + host-aware cross-repo tag moves (#1054) #1056)
• Blocker issue auto-filing for held promotions (feat(canary-rollout): auto-file blocker issues + rolling dashboard for held promotions (#1063) #1064)
• Cross-repo tag moves (canary-rollout PROMOTE fails for cross-repo (#482) agents: uses local 'git tag -f', not host gh-api (companion to #1049) #1054/feat: implement issue #1054 — canary-rollout PROMOTE fails for cross-repo (#482) agents: uses local 'git tag -f', not host gh-api (companion to #1049) #1055)
• Registry onboarding for cross-repo reusables (feat(canary-rings): onboard the 6 #482 reusables into the registry (#1036) #1044)

But the auto-promote gate is currently time-based (promote after observation window) with manual fleet-status review. The fleet monitor surfaces failure rates but doesn't feed them back into the promotion decision. Discussion #1042 (eval-score-gated promotion) addresses the pre-promote gate; this idea addresses the post-promote safety net — detecting regressions that slip through eval gating and manifest only under production traffic patterns.

Technical Opportunity

The fleet monitor (scripts/fleet_report.sh) already calculates per-workflow failure rates. The auto-promote script knows which agent version is on which ring. Connecting these — querying the fleet monitor's failure-rate data for the specific workflows running the promoted version, comparing against a baseline error budget — is architecturally clean.

The burn-rate SLO model provides a principled threshold:

• Error budget = allowed failures within the observation window (e.g., 5% over 48 hours)
• Burn rate = rate of error-budget consumption (e.g., 2x burn = budget exhausted in 24h instead of 48h)
• Rollback trigger = burn rate exceeds threshold for sustained period (not a single spike)

This avoids the noise problem of instantaneous failure-rate thresholds (one bad hour doesn't trigger rollback) while still catching genuine regressions within the observation window.

Assessment

Dimension	Score	Rationale
Feasibility	med	Requires connecting fleet-monitor data to canary-rollout decisions + defining the SLO schema; existing infra covers 70%
Impact	high	Closes the post-promotion safety gap for agents that modify their own infrastructure — the highest-stakes failure mode
Urgency	med	Auto-promote is already shipping (#1056); the safety gap grows as promotion automation matures

Adversarial Review

Strongest objection: Agent workflow failures are inherently noisy — a surge of complex PRs, a GitHub API outage, or a dependency update can spike failure rates without any issue in the agent version itself. Auto-rollback triggered by environmental noise wastes engineering time investigating phantom regressions and creates churn in the channel tags.

Rebuttal: The burn-rate SLO model specifically addresses noise. Unlike an instantaneous failure-rate threshold (which fires on any spike), the burn-rate model uses a time-windowed error budget. A single bad hour doesn't exhaust a 48-hour budget. The observation window size is the tuning knob — longer windows tolerate more noise but detect regressions slower. The fleet monitor's existing failure-rate data can be backtested against historical promotions to calibrate the SLO before enforcement. Implementation stages: (1) alert-only — post a comment on the promotion's blocker issue; (2) auto-hold — prevent the next scheduled promotion; (3) auto-rollback — move the channel tag back. Gate rollback behind a ROLLBACK_ENABLED repo variable (mirrors the LIVE_MODE pattern from premature-closure-audit.yml).

Suggested Next Step

Define the burn-rate SLO schema (error budget percentage, observation window, burn-rate threshold) as per-agent config in canary-rings.json, implement a post-promote health-check script that queries fleet-monitor data for the promoted version's workflows, and wire it into the scheduled auto-promote as an alert-only gate. Gate auto-rollback behind ROLLBACK_ENABLED repo variable — start with alerts, earn trust, then enable enforcement.