Repository navigation
Calibra v1.0.0: statistically rigorous LLM evaluation
Calibra checks four things a leaderboard number alone can't tell you: whether a gap between models is real, whether the rankings are stable, whether model confidences mean anything, and whether the test set leaked into training data. Every method is validated on synthetic data with a known ground truth.
Validation study (calibra study, re-run in CI)
- Paired bootstrap 95% CI: 0.945 coverage of the true accuracy gap.
- Exact McNemar: type-I error 0.042 at α=0.05, and power 0.851 for a 0.07 gap with n=400.
- Wilson 95% interval: 0.968 coverage.
- Bradley–Terry with Hunter MM: Spearman 1.000 against the true ratings, max error 23 Elo. Arena-style bootstrap CIs cover 7 of 8 true ratings.
- 2PL IRT: Spearman 0.991 for abilities θ and 0.959 for difficulties. Predicting full-benchmark accuracy from 30 anchors out of 600 items gives 38% lower error than a random 30-item subset (MAE 0.042 vs 0.068).
- Temperature scaling: fitted T = 2.41 against a true 2.5; ECE drops from 0.202 to 0.019.
- MinHash-LSH contamination: precision 1.000 and recall 0.975 on paraphrased leaks.
- Native C++20 kernels, bit-exact with their NumPy twins: bootstrap 10×, MinHash 3.1×.
Includes
- Paired significance tests: bootstrap, McNemar, sign-flip permutation.
- Benjamini–Hochberg FDR control and McNemar power analysis.
- Arena leaderboards with confidence intervals.
- IRT anchor-item benchmarking.
- ECE, Brier decomposition and temperature scaling.
- Contamination detection.
- Judge agreement: Cohen's κ, Krippendorff's α and a position-bias test.
- FastAPI service and CLI.
Quality gates (green Ubuntu CI)
- Tests pin literature reference values: Krippendorff α = 0.743 / 0.849, Wilson, κ, BH, FNV-1a and the closed-form Bradley–Terry result.
mypy --strict, ruff, 99.6% branch coverage, C++ built with-Werror.- No mocks.