Repository navigation
The first stable release. The Python API, the run.yaml and results.json formats, and the paperlab-space command now follow semantic versioning: configs and results from 1.0 keep working in every 1.x release.
- Text metrics:
bleuandchrf(chrF/chrF++) via sacreBLEU, with the signature saved for your paper, androuge1,rouge2,rougeL. Chinese, Japanese, Thai, Burmese, and other scripts without spaces are handled. - LLM-as-judge:
llm_judgegrades each output against the reference or your own rubric, with the same rate limits, retries, resume, and estimates as model requests. The judge's reasoning is saved for every row. - Confidence intervals: a 95% bootstrap interval for every score, in the Summary sheet, the CLI, and optionally
table.tex. - Significance tests: paired permutation (default) or bootstrap tests between models, Holm-corrected, in a new Significance sheet. With
statistics.baseline,table.texmarks significant differences with a dagger. results.jsonfiles from 0.1 still load. New dependencies:sacrebleuandnumpy.
Docs: LLM-as-judge · Statistics · Changelog