Repository navigation
v0.7.0
0.7.0 - 2026-10-03
Added
EvaluationSummary.metricsandEvaluationSummary.tags: every report
now breaks its results down by metric (MetricSummary: checks passed,
failed and skipped, pass rate, average score) and by case tag
(TagSummary: case counts, pass rate, average score).Score.metric: a stable key that reports group by. It defaults to
Score.name; the implicit checks whose names carry a per-case detail
(contains:<text>,required_field:<path>,required_tool:<tool>,
forbidden_tool:<tool>,tool_args:<tool>) share the metric named by
their prefix.Score.nameis unchanged.Score.skip(name, explanation)andScore.skipped: an evaluator can
record that a check did not apply to a case. A skipped score never
fails the case, is left out ofaverage_scoreand pass rates, and is
counted inMetricSummary.skipped.Eval()ignores a skippedScore
returned by a scorer.CaseEvaluation.tagsandCaseEvaluation.metadata, copied from the
TestCase, so a report can be filtered or regrouped on its own.GateConfig.min_tag_pass_rateandGateConfig.min_metric_pass_rate:
pass-rate floors for one tag or one metric, applied on top of
min_pass_rate. A tag or metric named in the config but absent from
the report fails the gate. Observed values are reported under
tag_pass_rate:<tag>andmetric_pass_rate:<metric>.TestCase.expected_tool_arguments: expected argument values per tool.
Passes when one call to the tool carries every listed argument with an
equal value (no type coercion). Scored astool_arg_values:<tool>.TestCase.required_tool_order: tools whose first calls must come in
the listed order. Scored astool_order.make_rubric(name, criteria, levels=..., with_reference=...): builds a
RubricTemplatefrom criteria written in plain language, with a
pass/fail scale by default or a caller-supplied graded scale.
All of the above are additive: reports saved by earlier versions still
load, and schema_version stays 1.0.
Fixed
evaluate_suiteno longer raisesZeroDivisionErrorwhen no case
produces a score (for example, every evaluator returned an empty list);
average_scoreis reported as0.0.