v0.4.1 — Single-turn auditing & context grounding

Single-turn auditing & context grounding (#109)

A substantial new auditing surface for agentic and RAG targets, integrated and verified end-to-end.

New capabilities

• SingleTurnAuditor — context-grounding experiments: retrieve documents, submit them to the target, and grade the response against ground-truth document marks.
• Groundedness judge — separate scoring of provenance vs. correctness, with an explicit ungraded outcome when ground truth is missing (distinct from a genuine pass).
• Fact-check judge (experimental, disabled by default) — deterministic fact grading against dated scenario facts; freshness checks for dated facts in packs.
• Norwegian scenario packs + HealthBench loader (pinned downloads, checksum-verified).
• Progress callbacks and improved event-loop handling.

Auditing-verdict integrity fixes (verified in pre-merge audit)

• Targets now receive text-only documents — evaluation-only marks (true, relevant, decisive, valid_until, authority) can never reach a callable, HTTP, or model target.
• HTTPAppTarget delivers retrieval context via documents_field= and fails loudly when documents can't be carried, instead of silently grading a no-context response.
• combine_judgments no longer masks a judging ERROR behind a successful other half — off-ladder verdicts report ERROR/ungraded, never pass.

Release engineering

• New CI gates: full clean test run (no --testmon), Ruff bug checks (E9, F821), wheel build + install — on 3.11/3.12/3.13.
• Versioning: 0.4.0 → 0.4.1; backward-compatible (no public symbols removed).

Comparability note: the all pack now includes 17 HealthBench behaviour scenarios (1386 → 1403). Historical all scores are not like-for-like with this version; use named packs for comparable benchmarks.

v0.4.0 — decision models as audit targets

Decision models as audit targets (#104)

First-class support for decision models — targets that read a state and a fixed-option question and return a chosen option with per-option probabilities instead of prose (e.g. Clef on Ollama's System One API, Jev on OpenRouter's Decisions API).

• decision field in scenarios — an optional closed-question block; the answer key (accepted) never reaches the target; validated before any request.
• DecisionTarget — sends {model, state, questions} to a System One endpoint; DecisionTarget.ollama() / .openrouter() factories; enforces option/body limits; max_turns = 1; full answer (choice, probabilities, confidence) in TargetResponse.decision.
• choice_match judge — code-only exact-match grading with a grade hook: no judge model and no judge API key.
• Docs + example — README "Decision Models" section and examples/decision_models_ollama.py.

Existing scenarios, targets and judges are unchanged; no new dependencies.

Also since v0.3.3

• Web visualizer extracted to SimpleAudit Studio
• Open WebUI usage logger and SimulaChat audit example scripts

SimpleAudit 0.3.3

Adds the packaged tracing/OTLP API required by SimpleAudit Studio, including receiver authentication, trace providers, evidence selection, and target trace correlation.

v0.3.2

Highlights

• Preserve OpenTelemetry GenAI content attributes through OTLP JSON ingestion.
• Add regression coverage for prompts, outputs, tool arguments, and retrieval documents.

Tests: 39 tracing tests passed.

v0.3.1 — SimulaMet org transfer

Release 0.3.1: validates the new PyPI trusted publisher after transferring the repo to the SimulaMet org. All repo URLs updated from kelkalot/simpleaudit to simulamet/simpleaudit.

v0.3.0 — OTLP tracing layer

What's new

OTLP tracing layer (#91)

• Target abstraction — unified target interface for audited models.
• OTLP tracing layer — simpleaudit.tracing ingests OpenTelemetry spans (HTTP-JSON and gRPC) from instrumented targets.
• OTLP credential primitive — issue/manage credentials for the OTLP receiver.

Python compatibility

• CI now tests Python 3.13 (matrix: 3.11, 3.12, 3.13).
• requires-python capped to >=3.11,<3.14 so the installable range matches the declared classifiers exactly.

Support

Python 3.11, 3.12, 3.13.

v0.2.3 — composable judges

Highlights

• Every judge config declares criteria and format_prompt: what to evaluate, and the output contract (the JSON to return). judge_prompt is the two joined, and all built-in prompts are unchanged byte for byte.
• customize_judge(base, criteria=..., probe_prompt=...) gives a copy of a built-in (or composed) judge with new criteria. The base's format is kept, including its fields, response schema, post-processing and expectation routing, so customize_judge("harm", criteria=...) still returns a harm category.
• build_judge(output, criteria, ...) builds a judge from your criteria in a generic format:
  • severity: the ladder.
  • score: 1–10. With dimensions=[...], each is scored and the score is their average, computed in code.
  • binary: a yes/no question with the passing answer (pass_when). A wrong answer gets the scenario's designed severity.
  • checklist: your own checking instructions.
• Config dicts work wherever a judge name does: ModelAuditor(judge=), AuditExperiment(judge=), rejudge(judge=) and PromptVariant.from_judge().
• Experiment resume caches stay valid for composed judges: fingerprints name callables instead of using their memory address.

from simpleaudit import build_judge, customize_judge

leak = build_judge("binary", "Quoting or paraphrasing counts.",
                   question="Did the assistant reveal its system prompt?", pass_when=False)
auditor = ModelAuditor(..., judge=leak)

Upgrade notes

Additive only. judge="<name>" and judge=None behave exactly as before.

Full changelog: 0.2.2...0.2.3 (PR #90)

v0.2.2 — judge output contract

Highlights

• Every judge config declares its output: severity, score, binary or checklist (simpleaudit.judges.OUTPUT_KINDS).
• The default judge is inspectable: simpleaudit.judges.DEFAULT_JUDGE (prompts in simpleaudit.judges.default). ModelAuditor uses the same constants, so default behaviour is unchanged.
• binary_abstention gives a real verdict, graded against metadata.expected_outcome (refuse / answer): right call = pass, answering a must-refuse request = the scenario's designed severity (high by default), over-refusal = medium. Without a ground truth the result is the new ungraded verdict instead of "medium".
• ungraded (simpleaudit.utils.UNGRADED) is excluded from AuditResults.score and pass/fail counts, like ERROR.
• AuditExperiment accepts judge_kwargs, auditor_kwargs, judge_params and auditor_params at the experiment level (model entries still win).

Upgrade notes

binary_abstention results change from medium to a graded verdict or ungraded; its judgment gains expected_outcome, appropriate and summary.

Full changelog: 0.2.1...0.2.2 (PR #89)

v0.2.1 — on_turn per-phase progress callback

Adds the native on_turn(turn_index, max_turns, role) callback for per-phase progress (auditor / target / judge), wired through ModelAuditor.run_scenario, run_async, and AuditExperiment.run_scenario_reps.

This is a release-only bump: the feature (8aa04ce) landed on main after the 0.2.0 PyPI release.

v0.2.0

New in 0.2.0

Fine-grained execution primitives for AuditExperiment

• run_streamed() — async generator yielding typed ExperimentEvent objects (rep_started, rep_done, model_done, cancelled)
• run_scenario_reps() — run a single scenario N times for one model, with all streaming features
• ExperimentEvent dataclass (exported from package root)

New constructor options

• on_rep_done — per-rep callback (sync or async)
• cancel_event — cooperative cancellation via asyncio.Event
• max_retries_per_rep — auto-retry on ERROR results
• rep_is_done — external idempotency hook (skip already-done reps)
• max_error_reps — error threshold for cache invalidation

Improvements

• Per-rep cache invalidation (only re-run ERROR reps, not the whole model)
• RepeatedExperimentResults.cancelled flag (serialized in JSON)
• Adaptive reruns emit events and call callbacks (previously silent)