💡 Pre-Merge AI Behavior Regression Testing Standard

[github-actions[bot]]

Summary

Define a standard for CI-integrated evaluation harnesses that catch prompt and model regressions in LLM-powered workflows BEFORE merge, using rubric-scored golden datasets rather than exact-match comparison. Complements the existing canary rollout system by adding a pre-merge quality gate for agent behavior.

Market Signal

Industry tools like Braintrust and Langfuse now run LLM evaluations directly in CI pipelines, so engineers can see whether a PR improves or degrades agent behavior before merging. 90% of developers use AI at work, but only 13% have AI operating across the full software delivery lifecycle — the evaluation gap in CI is the biggest barrier to trustworthy agent workflows. Enterprise ROI for agentic deployments averages 171%, but only when behavior regressions are caught early rather than discovered post-deployment.

User Signal

This org ships multiple LLM-powered reusable workflows: feature-ideation, pr-auto-review, ci-failure-analyst, and dev-lead. When prompts change or models are updated (e.g., Opus 4.5 → 4.6 → 4.7), the only quality gate is post-deploy canary failure accumulation. Issues #838 (pr-auto-review ring1→stable, cum_fail=5 REGRESSION) and #825 (feature-ideation next→ring0, cum_fail=2 REGRESSION) show canary blockers from regressions that could have been caught at the PR level — before they consumed canary budget and stalled the promotion pipeline.

Technical Opportunity

The canary rings system (scripts/lib/canary-rollout.sh) provides gradual deployment, but the gap is pre-merge. For each LLM-powered workflow, maintain a golden dataset of known-good inputs and rubric-scored expected behaviors (e.g., "output contains Assessment table," "cites at least one source," "proposes ≤ 5 ideas"). Run these as CI checks on PRs that modify prompt files, agent configurations, or model references. The existing bats test infrastructure can host evaluation harnesses — the pattern is identical to unit-testing pure decision scripts, already established in AGENTS.md.

Assessment

Dimension	Score	Rationale
Feasibility	med	Requires building eval harnesses per workflow and curating golden datasets. The bats infrastructure and CI template are ready.
Impact	high	Catches prompt regressions before canary, saving fleet-wide rollback costs and canary budget. Shifts quality left.
Urgency	med	Multiple agent workflows shipping and growing. Two active canary blockers (#838, #825) from behavioral regressions.

Adversarial Review

Strongest objection: LLM outputs are non-deterministic — golden datasets will be brittle and produce false positives, blocking legitimate PRs and creating developer friction.

Rebuttal: Use rubric-scored evaluation (does the output contain required sections? does it cite sources? is it within length bounds?) — not exact-match comparison. Structure tests as behavioral assertions, not string equality. Temperature=0 with structured output schemas provides determinism where needed. The canary system already accepts probabilistic quality; pre-merge eval adds an earlier, cheaper gate at the PR level. Start with the most deterministic workflows (compliance audit output, standards deploy decisions) where outputs follow a predictable schema, then expand.

Suggested Next Step

Select one LLM-powered workflow — pr-auto-review is a strong candidate since its output is structured and its canary has an active regression (#838). Build a minimal golden dataset of 5-10 test cases with rubric scoring (e.g., "output must contain a 'Recommendation' section," "output must not recommend merge when CI is red"). Wire it as a bats test that runs on PRs modifying the workflow's prompt or model reference. Measure the false-positive rate over 2 weeks before expanding to other workflows.