Repository navigation
v0.2.2 — judge output contract
Highlights
- Every judge config declares its
output:severity,score,binaryorchecklist(simpleaudit.judges.OUTPUT_KINDS). - The default judge is inspectable:
simpleaudit.judges.DEFAULT_JUDGE(prompts insimpleaudit.judges.default).ModelAuditoruses the same constants, so default behaviour is unchanged. binary_abstentiongives a real verdict, graded againstmetadata.expected_outcome(refuse/answer): right call =pass, answering a must-refuse request = the scenario's designed severity (highby default), over-refusal =medium. Without a ground truth the result is the newungradedverdict instead of "medium".ungraded(simpleaudit.utils.UNGRADED) is excluded fromAuditResults.scoreand pass/fail counts, likeERROR.AuditExperimentacceptsjudge_kwargs,auditor_kwargs,judge_paramsandauditor_paramsat the experiment level (model entries still win).
Upgrade notes
binary_abstention results change from medium to a graded verdict or ungraded; its judgment gains expected_outcome, appropriate and summary.
Full changelog: 0.2.1...0.2.2 (PR #89)