Skip to content

v0.4.0 — decision models as audit targets

Choose a tag to compare

@SushantGautam SushantGautam released this 09 Oct 11:25
· 167 commits to main since this release

Decision models as audit targets (#104)

First-class support for decision models — targets that read a state and a fixed-option question and return a chosen option with per-option probabilities instead of prose (e.g. Clef on Ollama's System One API, Jev on OpenRouter's Decisions API).

  • decision field in scenarios — an optional closed-question block; the answer key (accepted) never reaches the target; validated before any request.
  • DecisionTarget — sends {model, state, questions} to a System One endpoint; DecisionTarget.ollama() / .openrouter() factories; enforces option/body limits; max_turns = 1; full answer (choice, probabilities, confidence) in TargetResponse.decision.
  • choice_match judge — code-only exact-match grading with a grade hook: no judge model and no judge API key.
  • Docs + example — README "Decision Models" section and examples/decision_models_ollama.py.

Existing scenarios, targets and judges are unchanged; no new dependencies.

Also since v0.3.3

  • Web visualizer extracted to SimpleAudit Studio
  • Open WebUI usage logger and SimulaChat audit example scripts