Skip to content

v0.2.2 — judge output contract

Choose a tag to compare

@SushantGautam SushantGautam released this 28 Sep 13:04
· 195 commits to main since this release

Highlights

  • Every judge config declares its output: severity, score, binary or checklist (simpleaudit.judges.OUTPUT_KINDS).
  • The default judge is inspectable: simpleaudit.judges.DEFAULT_JUDGE (prompts in simpleaudit.judges.default). ModelAuditor uses the same constants, so default behaviour is unchanged.
  • binary_abstention gives a real verdict, graded against metadata.expected_outcome (refuse / answer): right call = pass, answering a must-refuse request = the scenario's designed severity (high by default), over-refusal = medium. Without a ground truth the result is the new ungraded verdict instead of "medium".
  • ungraded (simpleaudit.utils.UNGRADED) is excluded from AuditResults.score and pass/fail counts, like ERROR.
  • AuditExperiment accepts judge_kwargs, auditor_kwargs, judge_params and auditor_params at the experiment level (model entries still win).

Upgrade notes

binary_abstention results change from medium to a graded verdict or ungraded; its judgment gains expected_outcome, appropriate and summary.

Full changelog: 0.2.1...0.2.2 (PR #89)