Skip to content

v0.1.3: temperature-0 runs and robustness analyses

Choose a tag to compare

@rohithreddybc rohithreddybc released this 25 Sep 22:59

Adds temperature-0 replicates for the four locally served models, robustness analyses (paired bootstrap differences, leave-one-vignette-out, results excluding the undefined caution output, a bootstrap interval for the majority-vote reduction, a null-simulation sensitivity check, and a signed paired-test simulation), and single-sourced intervals at 5000 resamples. python harness/scripts/verify_paper_numbers.py checks 183 claims. The supplementary material is attached. Commit 80f6549.