EvalArc 0.16.0 — Decision coverage for recorded choices

Decision coverage for recorded choices

EvalArc can now show the trade-off between answering more decisions and making more mistakes. Give evalarc decision-coverage a file of labelled choice decisions, and it selects a confidence threshold using calibration records only. The report then shows coverage and error on held-out records. Provider errors and malformed responses remain visible as invalid rows, separate from wrong answers and abstentions.

python -m pip install evalarc==0.16.0
evalarc decision-coverage RECORDS.json --max-error 0.2 --output runs/decision-review

The new output directory contains a copy of the input, its SHA-256 digest, the full results in decisions.json, and a self-contained index.html. The guide and synthetic example explain the input format, result states, and exit codes.

The example data is synthetic. This command reviews recorded evidence; it does not calibrate confidence or establish a provider's accuracy. The new HTML report passed structure and escaping tests, but its layout and keyboard behavior have not been checked in a real browser.

The 24 new behavior tests, 514 full-suite tests, Ruff checks, site checks, and CI for Python 3.11–3.13 passed on the released source. PR #49 records the change and review. Prepared and reviewed with agent assistance; no human review is claimed.