Skip to content

V3.2.0 - Scoring accuracy & reliability

Latest

Choose a tag to compare

@n-papaioannou n-papaioannou released this 03 Jul 14:20

Scoring-accuracy & reliability release. Cuts false positives across the judge-path inspections and tightens the mandatory safety vetoes, with no CLI/API breaking changes.

Changed

  • Scoring engine. A judge that omits a mandatory dimension now retries → INCONCLUSIVE instead of a silent hard-veto FAIL (completeness guard). Non-mandatory presentation dimensions can no longer drag a correct response below threshold (shared _binary_score).
  • False-positive reduction across ~20 inspections. Rubrics now score the security/behavioral outcome, not presentation: a terse-but-correct response is no longer failed for a missing citation, rule ID, channel, or explanation.
  • B13 decision_attribution relaxed to trace-level — stops false-accusing complete audit trails that attribute their decisions, while still vetoing a fully unattributed trail.
  • B28 no_information_leak and B24 risk_proportionality are now mandatory vetoes — a real RAG leak or a "cry-wolf" over-rating now fails.
  • B24 min_evidence_items 20 → 12 (fewer INCONCLUSIVE on thin fixtures; wider CI band, documented in docs/scoring.md).
  • B23 reclassified to structural-only.
  • Plugin README now documents both Claude Code and Codex install.

Added

  • JSON-mode judge calls (json_output) for more reliable verdicts.
  • Vendor-aware self-judge detection (normalizes azure=openai, gemini=google).
  • Shared sample_capped (deterministic probe subsampling) and create_chat_completion_json_fallback provider helpers.

Removed

  • b23_policy_version_trace/rubric.yaml and references.yaml (structural-only).

Notes

  • Scores are recalibrated. Per-inspection and per-category scores from 3.2.0 are not directly comparable to 3.1.0 — re-run prior scorecards against the 3.2.0 harness before comparing. The overall-score normalizer is unchanged (no category added or removed).

Combines the scoring-engine and per-inspection reliability work (#60), replacing the earlier split PRs #58 / #59.