Skip to content

paired-eval 0.3.0

Choose a tag to compare

@alloevil alloevil released this 04 Sep 07:27
· 5 commits to main since this release

paired-eval 0.3.0

Added

  • gated verification class — the original principle in one construct: a programmatic (or any) gate runs first;
    failing it scores 0 and never calls the judge; only passing answers are scored by the score verification
    (typically a rubric). gate_min sets the pass line for continuous gates.
  • judge_check / bench_tasks: judge-scored tasks (rubric / gated / trajectory / retrieval) now flow through the
    same paired pipeline (run_interleavedreport), binarised at an explicit threshold.
  • Project website at https://alloevil.github.io/paired-eval/ (GitHub Pages from docs/): a single self-contained
    index.html generated by build_site.py — no external scripts, styles or fonts; zh/en toggle; the demo output,
    version and install command are pulled from the code, and tests/test_site.py fails if the file drifts from the generator.
  • Logo (docs/assets/logo.svg: two bars with confidence-interval whiskers) and a centered README header.

Changed

  • README and website reframed around the original question — evaluating models, agents and harnesses, verifying
    programmatically first and with rubrics only for what cannot be verified — instead of "compare two systems".
    The offline demo is now labelled as stubs standing in for models; the real-model quick start shows one A/B per
    object (model / harness / agent) on a task set that mixes exact, gated and trajectory classes.
  • README restructured after comparing nine reference projects (hypothesis, httpx, ruff, ragas, deepeval, openai/evals, …):
    install and a real example with output within the first 20 lines, no argumentative table; one-line GitHub
    description (203 → 115 chars, no jargon); homepage set.

Verified on this tag: 210 tests on Python 3.9 / 3.10 / 3.13 locally and 3.9 / 3.12 / 3.13 in CI; changed-line mutation ratchet 27/27 killed for the new verification code; every README code block executed against the real API; website generated from code and asserted equal to the committed file.