Bug: a run with zero cases reports passed and exits 0, so the gate silently disables itself

[royalpinto007]

What is wrong

A run with zero cases reports success. evalgate run suite.yaml --tags typo-here exits 0 with passed: true when the suite has no threshold.

src/runner.ts, runSuite (lines 557-578):

if (options.filterTags && options.filterTags.length > 0) {
  const wanted = new Set(options.filterTags);
  cases = cases.filter((c) => (c.tags ?? []).some((t) => wanted.has(t)));
}
...
const total = caseResults.length;
const passedCount = caseResults.filter((r) => r.passed).length;
const meanScore = total === 0 ? 0 : ...;
const thresholdMet = suite.threshold === undefined || meanScore >= suite.threshold;
const passed = thresholdMet && passedCount === total;

With total === 0: thresholdMet is true when no threshold is set, and passedCount === total is 0 === 0, so passed is true. cmdRun in src/cli/index.ts line 169 then returns 0.

Note the guard at line 572 shows the zero-case path was thought about for the mean score, but the pass verdict was not.

The interaction with compare is worse. An empty head run compared against a real baseline gives every case change: "removed" in compareRuns (src/compare.ts line 215), regressions is empty because regressions require both sides (line 209), so regressed is false and cmdCompare returns 0 (line 233). A misspelled tag, a renamed tag, or a suite whose cases all got filtered out turns the gate off completely and the CI job goes green.

That directly contradicts the tagline in HELP (line 106): "the build fails when your prompt gets dumber."

Why it matters

evalgate's entire job is to be the thing that says no. A silent no-op that reports success is the one bug class a CI gate cannot have, and suite.threshold being optional means plenty of real suites hit the vulnerable path. validateSuite in src/suite.ts already refuses an empty cases array (line 353), so an empty run is only reachable through filtering, which makes it easy to miss in testing.

Suggested approach

1. In src/runner.ts, make a zero-case run not pass. The cleanest version is an explicit total > 0 && in the passed expression, with a comment saying why.
2. Surface it. A RunResult with total: 0 should be visibly distinguishable in src/reporters/terminal.ts, not just a table with no rows.
3. Decide whether cmdRun should also warn on stderr when --tags matched nothing, naming the tags it looked for. That message is what turns a confusing red build into an obvious typo fix.
4. In src/compare.ts, consider whether an all-removed comparison should set regressed. It is a separate judgement call, so it is fine to leave it out and say so in the PR, but it should be a decision rather than an oversight.
5. Add tests: in tests/runner.test.ts, a suite run with a non-matching filterTags must not report passed: true; in tests/compare.test.ts, a head run with no cases against a populated baseline.

Comment below if you would like to take this one. I usually reply within a day.