Skip to content

v0.6.0 — Judge calibration, parallel reps, audit hardening

Choose a tag to compare

View all tags
@SushantGautam SushantGautam released this 06 Nov 17:10
· 16 commits to main since this release
v0.6.0
6718c26
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

What's new

  • Judge calibration (#145) — new simpleaudit.calibration module. label() collects human severity labels in the terminal; calibrate() scores a judge's stored verdicts against human labels or a judge from a different model family, with Wilson intervals, bias, a confusion table and severe misses. A same-family reference is refused unless allow_same_family=True. leaderboard(), save() and list_calibrations() included.
  • Agreement statistics (#145) — simpleaudit.stats.gwet_ac (Gwet's AC1/AC2) and krippendorff_alpha. Reports print the reference level counts, AC2 and alpha side by side; the leaderboard ranks by AC2, which stays reliable when most scenarios pass.
  • Parallel repetitions (#143) — AuditExperiment(max_parallel_reps=N) runs independent reps of run_scenario_reps concurrently (4 reps: 26 s → 7 s). Default 1 keeps the old behaviour.
  • Tracing evidence for the judge (#144) — OTLP spans keep status_message and exception events. summarize_for_judge shortens long values instead of cutting the span from the front, so every attribute stays visible, and shows status: ERROR / exception: lines for failed spans. New max_chars_per_value keyword.

Fixes and hardening (#143)

26 audit passes over grading, judges, targets, experiment caching, tracing and docs, each fix with a regression test.

  • Security: credentials redacted across span attributes and names, header text, reprs and truncated JSON, provider key formats, tool events and URLs, while the judge still sees the evidence it grades. OTLP auth fails closed. The auditor no longer inherits the judge's key across providers.
  • Grading: verdict parsing and severity normalisation hardened (think tags, stray keys, off-ladder values, contradictions, duplicate checklist entries). SingleTurnAuditor and ModelAuditor grade the same input the same way. Blank auditor probes are retried under max_retries.
  • Experiment cache: content-based fingerprints (judge logic, scenario text, dates, attachment bytes). Edited scenarios re-run; old runs are kept as .stale; resume re-runs only errored scenarios.
  • Clients: ModelAuditor.aclose(); owned HTTP clients are closed after each sync run() and each experiment rep.
  • Tracing: 16 MiB body cap on the HTTP and gRPC receivers; lowercase OTLP ids.

Behaviour changes

  • rejudge() now records the new judge in the result metadata and keeps the original under rejudged_from (previously it kept the source file's judge).
  • Single-turn transcripts carry the documents' text on the user turn.
  • Result JSONs add error_phase, parse_error and scenario_metadata; ModelStabilityReport.to_dict() adds n_ungraded.
  • The adaptive-settings sidecar is renamed adaptive-settings.json; saved files use the normal umask mode.

Process

Development now happens on dev; main only takes pull requests from dev for releases.

Full changelog: v0.5.1...v0.6.0

Assets 2
Loading