💡 Eval-Score-Gated Canary Promotion for Quality-Aware Progressive Delivery

[github-actions[bot]]

Summary

Enrich the canary-rollout promotion gate with agent output quality signals from the eval system, so promotions are gated not just on "did it crash?" (success/failure counts) but also "did quality degrade?" (eval score regression). This catches the class of failures where an agent produces worse output without hard-failing — the "silent quality regression" that the current pass/fail gate misses.

Market Signal

• FutureAGI canary deployment research: "When OpenAI shipped GPT-5, faithfulness improved at the 70th-percentile prompt but degraded at the bottom 5% — a stratified 1% canary surfaced this within hours." Binary pass/fail canary gates miss quality degradation in the long tail.
• Progressive delivery best practice now includes multi-signal canary gates incorporating quality metrics (not just crash metrics) into promotion decisions
• fast.io fleet management guide emphasizes "stratified health signals" — latency, timeout rate, fallback activation, AND quality score — for agent fleet promotion decisions
• The actions/checkout v7 security change (June 18, 2026) demonstrates how a behavioral change with no hard failures (PRs still build, tests still pass) can silently degrade agent workflows — exactly the class of regression that quality-aware gating catches

User Signal

• canary-rollout.sh (spec feat: implement issue #545 — Compliance: non-stub-dependency-audit.yml #548) currently evaluates success/failure counts with dwell times and sample targets — a binary "did it crash?" gate
• The eval system (💡 SkillOpt-Style Self-Improving Skills: Eval-Gated, Human-Reviewed Agent Skill Evolution #572, Skill eval health — triage (holdout) #998) already runs and produces scores for agent outputs, but those scores are not consumed by the promotion decision
• The eval health issues (Skill eval health — triage (holdout) #998, Skill eval infra error — triage (holdout) #988) show the eval pipeline is active but not yet connected to promotion
• The Opus 4.8 initiative ([Phase 1] Validate Opus 4.8 >= Opus 4.7 on the deep-review held-out eval suite (regression gate) #836: "Validate Opus 4.8 >= Opus 4.7 on the deep-review held-out eval suite") confirms that eval-based quality gating is already planned for model upgrades — but it's a one-off per-initiative gate, not wired into the generic canary promotion path
• The recent Sonnet 5 launch (June 30) creates an immediate evaluation need where quality-aware gating would add value

Technical Opportunity

The canary-rollout system is designed for extensibility:

• canary-rollout.sh is a pure-function library with no side effects — extending it with an eval-score input is a clean addition to the existing gate interface
• The gate spec (feat: implement issue #545 — Compliance: non-stub-dependency-audit.yml #548) is structured as per-transition criteria — adding an eval_score_delta criterion alongside the existing sample >= threshold criterion follows the established pattern
• The eval system produces scores that could be queryable via the same gh api calls used to gather run statistics
• Advisory-only introduction: emit the eval delta as a report field without making it blocking — mirrors the pattern used when the canary system itself was introduced (dwell-only → sample-gated → full gate)

Proposed gate extension interface:

# Existing gate (binary):
#   next→ring0: dwell >= 4h + sample >= clamp(round(0.25·avg), 3, 15)
#   ALWAYS: cumulative health = ZERO failures

# Extended gate (quality-aware):
#   next→ring0: dwell >= 4h + sample >= threshold + eval_delta >= -threshold%
#   eval_delta = (eval_score_candidate - eval_score_baseline) / eval_score_baseline
#   When eval sample < min_eval_sample: eval gate = SKIP (insufficient data)
#   When eval system errors: eval gate = WARN (no-signal, not block)

Assessment

Dimension	Score	Rationale
Feasibility	med	Requires connecting eval scores to canary gate — two systems that currently run independently. Starting advisory-only reduces integration risk.
Impact	high	Catches silent quality regression that passes the current binary gate. Every model upgrade (Sonnet 5, Opus 4.8, Gemini 3.5) benefits from quality-aware gating.
Urgency	low	The current binary gate works for hard failures. Quality-aware gating adds value for model upgrades, which happen periodically, not continuously.

Adversarial Review

Strongest objection: The eval system isn't stable enough to gate promotions. The eval health issues (#998, #988) suggest flakiness would block legitimate promotions if wired into the critical path.

Rebuttal: Start advisory-only: emit the eval score delta as a field in the canary report without blocking promotion. Promote to blocking only after the eval system proves stable over N promotion cycles. Require a minimum eval sample size before the score delta is meaningful, and treat eval-system errors as "no signal" (WARN) rather than "block." This follows the same graduated-enforcement pattern the canary system itself used (dwell-only → sample-gated → full gate).

Suggested Next Step

Define the eval-score interface for canary-rollout.sh: an optional eval_score_before / eval_score_after pair per transition, with a configurable regression threshold and minimum sample size. Implement as advisory-only initially — the gate reports the delta but does not block promotion.