v0.43.0-beta

Warning

This is a preview release. APIs may change without notice. Not for production use.
See DISCLAIMER.md for full details.

No verdict reads PASS when something failed or was not measured

One release with three parts:

• no verdict reads PASS when something failed or was not measured (reported in #203, #279);
• no real target, no measurement: the bench commands, redteam, eval, the samples and the judges stop standing in for a model, and a check that could not run stops reading as a pass (#275);
• over-refusal graded by a judge, grader test sets that agree with their thresholds, and calibration reports in the workspace folder (#274).

Upgrading from 0.42

Runs that passed before can now read WARN (exit 10), FAIL (exit 9) or INCONCLUSIVE, each because something failed or was not measured that the old verdict hid. Decide how your CI treats exit 10. bench, redteam and eval need a real target, AGENTEVAL_ALLOW_STUB_JUDGE is ignored, and a command line that does not parse exits 2.

Highlights

• A part that didn't run never leaves a clean PASS. A composite with a required component that was not measured reads WARN, at every level, and so do benchmark runs with unmeasured rows, bench gdpr --runs N with runs that gave no verdict, and bench longmemeval with unscored questions. A red-team attack that measured nothing, or a run with more inconclusive probes than resisted ones, reads INCONCLUSIVE and withholds the OWASP, MITRE and NIST pass. A run where nothing was measured is never a pass.
• A failing check doesn't hide under an average. In the agentic presets and the seven evaluators built from sub-dimensions, a failing accuracy dimension fails the verdict and a failing quality dimension makes it WARN, naming it. GDPR and EU AI Act Standard and Smoke fail on a high or critical article failure and warn on a medium one.
• Judges see what they grade. Each agentic judge gets its own rubric as the system prompt and its verdict comes from that rubric's bands; with --trace, the tool checks receive the run's tool calls and definitions.
• Not measured is not a fail. The older metrics (faithfulness, context precision and recall, answer correctness, the embedding metrics, MRR, Recall@K, the Microsoft.Extensions.AI adapters) report "not measured" when an input they need is missing, instead of failing at 0. Reports and exports say so, and the MAF bridges pass the reference answer and retrieved context along.
• Calibration reports only measured verdicts. A judge outage is INFRA-FAIL; an evaluator not measured on every record is INCOMPLETE. Figures for one judge are in Calibration results: GDPR 5 of 6 pillars, EU AI Act 5 of 6, agentic 5 of 8 categories (process fails at 61.5%, κ 0.323).
• A real model or stop. The bench commands no longer measure built-in stand-ins, the stub judge is retired, the Getting Started and Gatekeeper samples run on the configured model, bench autoaudit audits real models, and the OWASP, MITRE and NIST benchmarks call their judge. A regex timeout or a cost missing from the price table reads "could not check", not a pass.
• Gatekeeper v1 fixes. The startup check sees every judge, and coverage never reads as full when nothing was measured.
• Over-refusal is graded by a judge, not a phrase list; grader test sets agree with their thresholds; calibration reports go to .agenteval/.
• A healthcare safety domain pack, contributed by @goktugozkanmd (#273): five checks for synthetic healthcare-support cases, a real model by default, judges calibrated against the author's gold labels with --calibrate, offline tests in CI. Sample O1 in AgentEval.Samples (#282) runs three cases live, then the same checks on three canned unsafe replies as controls. Not clinical validation.

Full details, with the cause and evidence for every change: CHANGELOG.