Skip to content

v0.13.0-beta — Judge-primary grading + Composite Judges

Pre-release
Pre-release

Choose a tag to compare

@joslat joslat released this 23 Jun 22:43
· 195 commits to main since this release

Red-team grading: judge-primary by default + Composite Judges (ADR-021→024)

The red-team grader — which decides whether each attack succeeded — moves from a keyword/substring oracle to LLM-judge-primary grading with honest-by-construction "Composite Judges." Each verdict is split into a positive-only compromise detector ⊕ a negative-only refusal detector, each structurally clamped so it can only raise its own direction or abstain. Driven by a reproducible finding that iteratively patching a keyword oracle does not converge to an honest grader.

⚠️ Breaking change (default behavior)

  • --judge-mode now defaults to primary (was fallback). With a judge configured, Composite Judges lead the verdict. A run with no --judge stays byte-identical to the keyword oracle, and --judge-mode fallback restores the prior behavior.
  • Default judge rubric is now evidence-anchored (was strict).

Added

  • Composite Judges for all six semantic oracles (InferenceAPIAbuse, InsecureOutput, SupplyChain, DataPoisoning, Misinformation, ExcessiveAgency).
  • ADR-024 split-then-gate gated tree for InferenceAbuse — closes the "I am Grok" model-vs-product-bot floor on gpt-4o-mini with no stronger model; documented bound (it regresses InsecureOutput, so it's applied only where it helps).
  • Reasoning-model judges (o-series / GPT-5-class) — auto-retry without temperature.
  • --judge-timeout now bounds the composite path; AgentEval.SampleGraders companion.

Verification

κ=1.000 (n=92), 0 directional fabrications over the 314-case adversarial corpus; K=10 stochastic = 810 trials / 0 fabrications; full suite green (net8/9/10).

Migration: add --judge-mode fallback to keep the prior default grading; no change needed for no-judge runs.

Full details: CHANGELOG.md · ADR-021/022/023/024.