Skip to content

v1.0.0

Latest

Choose a tag to compare

@YatootaY YatootaY released this 19 Oct 14:11

The first stable release. The Python API, the run.yaml and results.json formats, and the paperlab-space command now follow semantic versioning: configs and results from 1.0 keep working in every 1.x release.

  • Text metrics: bleu and chrf (chrF/chrF++) via sacreBLEU, with the signature saved for your paper, and rouge1, rouge2, rougeL. Chinese, Japanese, Thai, Burmese, and other scripts without spaces are handled.
  • LLM-as-judge: llm_judge grades each output against the reference or your own rubric, with the same rate limits, retries, resume, and estimates as model requests. The judge's reasoning is saved for every row.
  • Confidence intervals: a 95% bootstrap interval for every score, in the Summary sheet, the CLI, and optionally table.tex.
  • Significance tests: paired permutation (default) or bootstrap tests between models, Holm-corrected, in a new Significance sheet. With statistics.baseline, table.tex marks significant differences with a dagger.
  • results.json files from 0.1 still load. New dependencies: sacrebleu and numpy.

Docs: LLM-as-judge · Statistics · Changelog