multivon-eval 0.16.0 — reliability metrics, grader validation, judge integrity

The consensus wave: designed by a two-track multi-LLM deliberation (7 personas × Claude/GPT-5.6/Gemini-3.1 democratic ballots + blind independent model reviews with cross-examination), gated by a three-model verification that voted round 1 unanimously not ready (4 majors, all fixed with ~40 tests) and round 2 unanimously ready with zero new majors.

Added

• pass@k / pass^k from your existing --runs N data: unbiased estimators (never the (c/n)^k plug-in), cluster-bootstrap CIs that resample cases not trials, honest UNKNOWN when k > runs — no extrapolation. report.pass_hat_k(k), lottery cases, assert_pass_hat_k CI-lower-bound gate.
• multivon-eval validate — grade your graders: run your evaluators against each task's reference output; a failing reference means a broken task or grader, not a bad model. Zero-pass suspects, contrast-twin discrimination, never calls the model under test.
• Saturation monitor: a 100% suite can't detect improvement — the report now quantifies the Wilson floor + minimum detectable regression and recommends graduating to a regression suite (EvalSuite(purpose=...)).

Fixed (score integrity)

• Hedged judge replies ("I cannot say yes or no…") no longer parse as YES; unparseable verdicts are UNKNOWN, excluded and disclosed; low verdict coverage raises JudgeUnavailable.
• Judge errors no longer launder into 0.0 quality scores — they surface as error statuses excluded from pass rate.
• Error budget on the CI gate (max_error_rate): 90 errored cases + 10 passes can no longer show a green 100% gate.
• Explicit JudgeConfig(temperature=0.0) now actually overrides a nonzero global.

Full details: CHANGELOG · guides: reliability-metrics · validate · Demystifying evals, operationalized