Repository navigation
Single-turn auditing & context grounding (#109)
A substantial new auditing surface for agentic and RAG targets, integrated and verified end-to-end.
New capabilities
SingleTurnAuditor— context-grounding experiments: retrieve documents, submit them to the target, and grade the response against ground-truth document marks.- Groundedness judge — separate scoring of provenance vs. correctness, with an explicit
ungradedoutcome when ground truth is missing (distinct from a genuine pass). - Fact-check judge (experimental, disabled by default) — deterministic fact grading against dated scenario facts; freshness checks for dated facts in packs.
- Norwegian scenario packs + HealthBench loader (pinned downloads, checksum-verified).
- Progress callbacks and improved event-loop handling.
Auditing-verdict integrity fixes (verified in pre-merge audit)
- Targets now receive text-only documents — evaluation-only marks (
true,relevant,decisive,valid_until,authority) can never reach a callable, HTTP, or model target. HTTPAppTargetdelivers retrieval context viadocuments_field=and fails loudly when documents can't be carried, instead of silently grading a no-context response.combine_judgmentsno longer masks a judgingERRORbehind a successful other half — off-ladder verdicts reportERROR/ungraded, neverpass.
Release engineering
- New CI gates: full clean test run (no
--testmon), Ruff bug checks (E9, F821), wheel build + install — on 3.11/3.12/3.13. - Versioning: 0.4.0 → 0.4.1; backward-compatible (no public symbols removed).
Comparability note: the all pack now includes 17 HealthBench behaviour scenarios (1386 → 1403). Historical all scores are not like-for-like with this version; use named packs for comparable benchmarks.