One provider or scorer error discards the entire run

[royalpinto007]

Problem

One failed provider call throws away the entire run.

runCase in src/runner.ts calls the provider with no error handling:

const response = await provider.complete(toRequest(c, model));

mapPool awaits fn inside its workers and runSuite awaits Promise.all(workers), so a throw from any single case rejects all the way out of runSuite. The CLI catches it, prints a message, and exits. Every case that already completed is discarded: no artifact is written, no partial report, nothing to compare against.

The throws are not exotic. OpenAICompatibleProvider.complete in src/providers/openai-compatible.ts throws on any non-2xx response:

if (!res.ok) {
  const text = await res.text().catch(() => "");
  throw new Error(`[evalgate] ${this.name} request failed (${res.status}): ${text}`);
}

A 429 is the single most likely response when running a suite of cases against an LLM API, and with --concurrency above 1 it is close to guaranteed. A scorer can throw too: see #8, where an invalid pattern in a JSON schema kills the whole run.

Why it matters

This is a CI tool. A 40 case suite that dies on case 37 because of one rate limit response has burned 37 API calls, spent real money, and produced nothing, and the failure looks identical to a genuine eval failure. Worse, the natural user response to a flaky gate is to stop trusting it.

Suggested approach

1. Catch per case. Wrap the provider call and each scorer call in runCase, and on failure return a CaseResult with passed: false, score: 0, and a new error?: { phase: "provider" | "scorer"; message: string } field. The run continues.
2. Distinguish "failed the eval" from "could not be evaluated" everywhere downstream. RunResult should carry an errored count, the reporters should show it separately from failures, and compareRuns should not treat an errored case as a genuine score regression, because a rate limit is not a prompt getting dumber.
3. Decide the exit code deliberately: a run containing errored cases should not exit 0, but it should say why in different words than a threshold failure.
4. Coordinate with the sibling issue about timeouts and retries. Retrying transient failures is the first line of defense, and per-case error capture is the second. Both are needed.
5. Tests: a provider that throws on the third case of five, a scorer that throws, and an assertion that the artifact still contains all five cases with the right shapes.

Done when

• A single provider or scorer failure never discards completed results.
• Errored cases are visible and counted separately from failed ones.
• The artifact is always written.

---

If you want to take this on, comment on the issue to claim it and it will be assigned. Please keep to a maximum of 2 open claims per person at a time so other contributors get a chance.