💡 Canary Failure Auto-Triage & Diagnostic Recovery Standard

[github-actions[bot]]

Summary

Enhance the canary rollout engine to automatically classify failures (transient vs. persistent vs. regression), auto-retry transient failures, and generate rich diagnostic context for persistent failures — reducing the human toil of managing canary blockers while preserving the human gate for promotion decisions.

Market Signal

Enterprise agentic CI ROI is highest in incident response and self-healing (171% average ROI per industry reports). The industry trend toward self-healing pipelines focuses on automated triage and recovery, not automated bypass. GitHub's own 2026 security roadmap emphasizes "secure by default" — canary systems should fail fast with actionable context, not silently accumulate opaque failure counts. The distinction between auto-triage (automated diagnosis) and auto-heal (automated resolution) is becoming a recognized best practice in platform engineering.

User Signal

Five open canary blocker issues demonstrate the current pain:

• Canary blocker: pr-auto-review ring1->stable (cum_fail=1, REGRESSION) #838: pr-auto-review ring1→stable (cum_fail=5, REGRESSION)
• Canary blocker: feature-ideation next->ring0 (cum_fail=2, REGRESSION) #825: feature-ideation next→ring0 (cum_fail=2, REGRESSION)
• Canary blocker: dependabot-automerge ring0->ring1 (cum_fail=1, REGRESSION) #822: dependabot-automerge ring0→ring1 (cum_fail=1, REGRESSION)
• Canary blocker: add-to-project ring0->ring1 (cum_fail=27, PRE_EXISTING) #701: add-to-project ring0→ring1 (cum_fail=27, PRE_EXISTING)
• Canary blocker: ci-failure-analyst next->ring0 (cum_fail=0, -) #732: ci-failure-analyst next→ring0

These blockers stall the entire promotion pipeline. Most carry needs-human labels — but when humans arrive, they find minimal diagnostic context beyond a cumulative failure count. The gap is not detection (the canary system catches failures) but triage (classifying what kind of failure and providing actionable next steps).

Technical Opportunity

The canary rollout engine (scripts/lib/canary-rollout.sh) already classifies failures and accumulates counts. Issue #803 (hardening _run_json with exponential jittered backoff) demonstrates the pattern of adding retry logic at the API call level. Extend this to the canary decision level:

1. Classify each failure as transient (API timeout, rate limit, runner OOM), persistent (same test fails on every run), or regression (test that previously passed now fails)
2. Auto-retry transients with exponential backoff (up to 3 attempts)
3. Enrich persistent/regression failures with diagnostic context: diff of what changed since last green run, link to failing logs, list of modified files
4. Escalate with structured data rather than opaque cum_fail=N labels

Assessment

Dimension	Score	Rationale
Feasibility	med	Requires enhancing the canary-rollout engine's failure classification logic. The retry and diagnostic patterns already exist in the codebase (#803).
Impact	high	Unblocks stalled promotions, reduces human triage toil, and provides actionable recovery context. Directly addresses 5 open blocker issues.
Urgency	high	Five open canary blocker issues right now. #701 has been accumulating failures for weeks (cum_fail=27). The promotion pipeline is materially stalled.

Adversarial Review

Strongest objection: Auto-retrying canary failures masks real issues. The whole point of canaries is to catch problems — adding self-healing defeats the purpose and could promote broken workflows to production.

Rebuttal: The proposal is auto-TRIAGE, not auto-PROMOTE. The human gate for promotion decisions is preserved — only the diagnosis is automated. Transient failures (API timeouts, runner capacity) are already known false signals; retrying them automatically reduces noise without masking real issues. Persistent failures get RICHER context, not less. The #803 issue already added exponential backoff at the API call level — this extends the same well-tested principle to the canary decision layer. The cum_fail=27 PRE_EXISTING label on issue #701 shows failures accumulating for weeks without resolution — automated triage would have classified these and surfaced an actionable remediation path weeks ago.

Suggested Next Step

Extend scripts/lib/canary-rollout.sh to classify each failure into transient/persistent/regression categories based on exit code, HTTP status, and failure pattern matching. For transient failures (HTTP 5xx, timeout, rate-limit), auto-retry up to 3 times with jittered backoff. For persistent/regression failures, attach a diagnostic payload to the canary blocker issue body: git diff between last green SHA and current SHA, link to the most recent failing workflow run, and list of files changed. Pilot with dependabot-automerge (the simplest canary with an active blocker #822).