Skip to content

paired-eval 0.2.0

Choose a tag to compare

@alloevil alloevil released this 04 Sep 05:32
· 9 commits to main since this release

paired-eval 0.2.0

Added

  • paired_eval module: a single import surface matching the repository name (import paired_eval as pe), re-exporting
    the public API of the implementation modules, plus pe.demo() / python3 paired_eval.py — an offline demo that
    prints a complete report from two stub systems in under a second.
  • claim_eval.fmt_p: report p-values below 1e-4 in scientific notation instead of 0.0000.
  • Report text in English: set_language("en") (or lang="en" on interpret / report, --lang en on the demo).
    The Chinese strings are byte-identical to before; a test asserts the English output contains no CJK
    characters and that both languages expose the same placeholders.

Changed

  • README rewritten from the user's perspective (why / install / ten-second demo / your own tasks / how to read the
    report / API / scope). The demo output pasted in both READMEs is asserted equal to the actual output by a test, and
    every code block is executed against the real API.
  • README gains a flow diagram and a grounded "how it relates to other tools" section (lm-evaluation-harness,
    Inspect, promptfoo, openai/evals — descriptions taken from their own READMEs).

Verified on this tag: full fast suite on Python 3.9 / 3.10 / 3.12 / 3.13 (locally and in CI), hook integration, power calibration, hand-picked mutants; changed-line mutation ratchet 4/4 killed.