💡 Canary Pipeline Self-Health Observability & Reliability Standard

[github-actions[bot]]

Summary

The canary-rollout system has been the source of 6+ bugs in a single month (Aug-Sep 2026: #1066, #1065, #1057, #1049, #1047, #1046), all stemming from emergent interactions between autocut, promote-all, ship-drift, and the evaluation gate. This proposes a standard for monitoring the canary system's own operational health — separate from the individual agent promotions it manages — including structured health scores, interaction invariant checks, and alerting when the promotion pipeline itself is degraded.

Market Signal

Industry trend toward "observability of observability" — monitoring the monitoring systems themselves. GitHub's Actions Data Stream (2026 roadmap) recognizes that CI pipelines need first-class telemetry. The proliferation of agentic CI automation makes self-monitoring essential: when automation manages automation, silent degradation is the default failure mode.

User Signal

• 6+ canary-rollout bugs in Aug-Sep 2026: Autocut runs before promote-all in the same job, so a busy agent's dwell resets every tick and can never promote — dev-lead stuck at next since 08-12 (blocks #1592) #1066, canary-rollout ring promotion can skip the v-scoped channel tag, breaking every stub deployed to that tier #1065, canary gate counts engine-quota failures (Claude 429 session limit + dead Copilot fallback) as cum_fail — dev-lead next->ring0 is on a false REGRESSION; blocks #1592 #1057, ship-drift reports '0 agents merged-but-unshipped' while all 7 dev-lead stubs are 29-181 commits behind — it compares the 'next' channel, which no stub pins (the #1592 detector is blind to #1592) #1049, canary gate counts a concurrency-EVICTED run (cancelled, 0 steps executed) as cum_fail — dev-lead next->ring0 is sitting on a false REGRESSION recommending rollback; blocks #1592 #1047, canary gate: a v-scoped channel tag shadows the release tag in _gh_candidate_cut_date, so any cross-repo agent adopting v<M>-<tier> channels is permanently BLOCKED (indeterminate) — blocks #1592 #1046
• Shared pattern: independent automations interact unexpectedly — autocut resetting dwell timers before promote-all (Autocut runs before promote-all in the same job, so a busy agent's dwell resets every tick and can never promote — dev-lead stuck at next since 08-12 (blocks #1592) #1066), ship-drift comparing the wrong channel (ship-drift reports '0 agents merged-but-unshipped' while all 7 dev-lead stubs are 29-181 commits behind — it compares the 'next' channel, which no stub pins (the #1592 detector is blind to #1592) #1049), compliance-audit closing live findings (compliance-audit.sh closes live findings: fail-open closure, umbrella title mismatch, no execution guard #1036)
• dev-lead was stuck at "next" since 08-12, blocking #1592 — the system had no mechanism to surface this stall
• The canary pipeline is the critical path for ALL agent deployments

Technical Opportunity

The canary-rollout infrastructure already emits structured data (evaluate output, ring state, channel tags). A health-score layer can aggregate this into a single "canary pipeline health" signal — checking invariants like:

• promote-all runs before autocut (prevents dwell-timer reset)
• ship-drift compares the correct channel per agent's actual pin
• No agent has been stuck at the same ring for > N days without a blocker issue
• cum_fail counts exclude known benign categories (concurrency evictions, engine-quota 429s)

The daily-org-status report (#1068 pattern) provides a natural integration point.

Assessment

Dimension	Score	Rationale
Feasibility	high	Builds on existing canary data; invariant checks are straightforward assertions over existing state
Impact	high	Directly prevents the bug class that has dominated the issue tracker for a month; reduces human triage effort
Urgency	high	Canary bugs are the #1 operational issue right now; dev-lead promotion has been blocked for 3+ weeks

Adversarial Review

Strongest objection: Adding monitoring to a system that's already complex risks making it more complex. The real fix is to fix the individual bugs, not to monitor for new ones.

Rebuttal: The bugs keep recurring because the interaction patterns are non-obvious — each script works correctly in isolation but fails when composed. An invariant-checking layer catches NEW interaction bugs proactively, before they block promotions for weeks. The health check IS the regression test suite for the canary system itself, which currently has none. Fixing bugs one at a time is necessary but insufficient when the system generates new interaction failures faster than they're fixed.

Suggested Next Step

Define the invariant checklist (ordering constraints, channel-comparison correctness, dwell-timer monotonicity) and implement as a post-canary-rollout diagnostic step that writes to the daily org status report. Start with the three invariants that would have caught #1066, #1049, and #1036.