paired-eval 0.3.0
paired-eval 0.3.0
Added
gatedverification class — the original principle in one construct: a programmatic (or any)gateruns first;
failing it scores 0 and never calls the judge; only passing answers are scored by thescoreverification
(typically a rubric).gate_minsets the pass line for continuous gates.judge_check/bench_tasks: judge-scored tasks (rubric / gated / trajectory / retrieval) now flow through the
same paired pipeline (run_interleaved→report), binarised at an explicit threshold.- Project website at https://alloevil.github.io/paired-eval/ (GitHub Pages from
docs/): a single self-contained
index.htmlgenerated bybuild_site.py— no external scripts, styles or fonts; zh/en toggle; the demo output,
version and install command are pulled from the code, andtests/test_site.pyfails if the file drifts from the generator. - Logo (
docs/assets/logo.svg: two bars with confidence-interval whiskers) and a centered README header.
Changed
- README and website reframed around the original question — evaluating models, agents and harnesses, verifying
programmatically first and with rubrics only for what cannot be verified — instead of "compare two systems".
The offline demo is now labelled as stubs standing in for models; the real-model quick start shows one A/B per
object (model / harness / agent) on a task set that mixesexact,gatedandtrajectoryclasses. - README restructured after comparing nine reference projects (hypothesis, httpx, ruff, ragas, deepeval, openai/evals, …):
install and a real example with output within the first 20 lines, no argumentative table; one-line GitHub
description (203 → 115 chars, no jargon); homepage set.
Verified on this tag: 210 tests on Python 3.9 / 3.10 / 3.13 locally and 3.9 / 3.12 / 3.13 in CI; changed-line mutation ratchet 27/27 killed for the new verification code; every README code block executed against the real API; website generated from code and asserted equal to the committed file.