Skip to content

v0.4.1 — Single-turn auditing & context grounding

Latest

Choose a tag to compare

@SushantGautam SushantGautam released this 10 Oct 21:24
· 13 commits to main since this release

Single-turn auditing & context grounding (#109)

A substantial new auditing surface for agentic and RAG targets, integrated and verified end-to-end.

New capabilities

  • SingleTurnAuditor — context-grounding experiments: retrieve documents, submit them to the target, and grade the response against ground-truth document marks.
  • Groundedness judge — separate scoring of provenance vs. correctness, with an explicit ungraded outcome when ground truth is missing (distinct from a genuine pass).
  • Fact-check judge (experimental, disabled by default) — deterministic fact grading against dated scenario facts; freshness checks for dated facts in packs.
  • Norwegian scenario packs + HealthBench loader (pinned downloads, checksum-verified).
  • Progress callbacks and improved event-loop handling.

Auditing-verdict integrity fixes (verified in pre-merge audit)

  • Targets now receive text-only documents — evaluation-only marks (true, relevant, decisive, valid_until, authority) can never reach a callable, HTTP, or model target.
  • HTTPAppTarget delivers retrieval context via documents_field= and fails loudly when documents can't be carried, instead of silently grading a no-context response.
  • combine_judgments no longer masks a judging ERROR behind a successful other half — off-ladder verdicts report ERROR/ungraded, never pass.

Release engineering

  • New CI gates: full clean test run (no --testmon), Ruff bug checks (E9, F821), wheel build + install — on 3.11/3.12/3.13.
  • Versioning: 0.4.0 → 0.4.1; backward-compatible (no public symbols removed).

Comparability note: the all pack now includes 17 HealthBench behaviour scenarios (1386 → 1403). Historical all scores are not like-for-like with this version; use named packs for comparable benchmarks.