Deleting a failing case makes the gate green: removed cases are never regressions

[royalpinto007]

Problem

Deleting a case is a way to make the gate green.

compareRuns in src/compare.ts classifies a case present in the baseline and absent from the candidate as "removed":

} else if (!b && h) {
  change = "added";
} else {
  change = "removed";
}

and the verdict only counts "regressed":

const regressions = cases.filter((c) => c.change === "regressed");
...
regressed: regressions.length > 0,

So a contributor whose prompt change breaks the handles_refund_request case can delete that case from the suite and compareRuns reports no regressions at all. Nothing warns, nothing fails, and the PR comment written by src/github.ts says the run is clean.

There is a second, quieter version of the same problem. overallDelta is head.score - baseline.score, and score is the mean over cases. Removing the lowest scoring cases raises the mean, so a suite that shrinks looks like a suite that improved.

Related: #6 notes that a run with zero cases reports passed and exits 0. That is the extreme end of this same hole. Deleting every case is just the limiting case of deleting one.

Why it matters

This is the core value proposition. The README says the build fails when your prompt gets dumber. A regression gate that can be silenced by editing the test suite in the same commit is not a gate, and this is the first thing a reviewer of this project will try.

Suggested approach

1. Add a policy for removed cases, defaulting to strict. Options worth exposing, in CompareOptions and as a CLI flag:
   • error (suggested default): any case in the baseline and missing from the candidate makes regressed true.
   • warn: reported prominently in the terminal, markdown, and JUnit reporters but does not fail.
   • allow: current behavior, for the legitimate case of intentionally retiring a case.
2. Report removals explicitly. Add a removals: CaseDelta[] field next to regressions and improvements, and render it in every reporter. Right now removals are computed and then thrown away.
3. Fix the mean comparison. Either compute overallDelta over the intersection of case ids, so the two means are over the same set, or report both the intersection delta and the full-suite delta and label them. Silently comparing means over different populations is the kind of thing this tool exists to catch in other people's systems.
4. Consider checking RESULT_VERSION compatibility between the two artifacts while you are here. compareRuns takes two RunResults and never looks at version.
5. Tests: baseline with 3 cases versus candidate with 2, with and without each policy, and a case where removing the worst case raises the mean.

Done when

• Removing a case from the suite cannot silently produce a clean verdict.
• Removals appear in every reporter.
• The overall delta compares like with like.
• The policy is documented in the README next to the tolerance option.

---

If you want to take this on, comment on the issue to claim it and it will be assigned. Please keep to a maximum of 2 open claims per person at a time so other contributors get a chance.