A judge-integrity patch. Non-reasoning judges are byte-identical to 0.16.0; if you point a gpt-5.x or o-series model at the QAG evaluators, this is the release that makes it work.
Reasoning-tier judges no longer truncate before emitting the verdict
The judge calls hardcoded small per-call token caps — 100 for a yes/no verdict, 512 for claim extraction. That is ample for a plain-text judge. A reasoning-tier model spends part of its output budget on hidden reasoning before the verdict, so those caps cut it off mid-think and it returned nothing.
We measured this at a 47.3% error rate with gpt-5.5 as the judge, in our own preregistered study's strong-judge ablation — and published it against ourselves under the preregistered errors-as-failures rule rather than quietly repairing it. This release is the repair.
The effective per-call ceiling is now floored at 2048 tokens for reasoning-tier judges, detected via the same gpt-5/o1/o3/o4 prefix signal the rest of the SDK already uses. Non-reasoning judges (gpt-4o-mini, claude-haiku-4-5, local models) are byte-identical — their cap passes through untouched, and an explicit larger max_tokens is never lowered. The QAG prompt and parser are unchanged, so calibrated thresholds still hold.
Also in this release
multivon-eval validateon the shippedquickstarttemplate now exits 0. The scaffolded cases had no reference output, so 0.16.0's flagship command's first contact was a red ✗.- The
python -m multivon_evaldemo no longer prints a rawUserWarningabove the banner. The uncalibrated-threshold advisory now routes through the demo's own⚠channel. - README disagreement headline is sourced to the committed benchmark. It now leads with Cohen's κ ≈ 0.04 on the binary hallucination verdict and the RAGTruth-Sum flip counts that trace to
eval-framework-benchmark, replacing an unsourceable figure.
Full detail: CHANGELOG
pip install --upgrade multivon-eval