v0.4.0

What's Changed

其他 / Other

• feat: paired_eval package, PyPI release, English-first README by @alloevil in #3
• feat: AgentXRay adapter — first real path onto the agent axis by @alloevil in #4

New Contributors

• @alloevil made their first contribution in #3

Full Changelog: v0.3.0...v0.4.0

paired-eval 0.3.0

paired-eval 0.3.0

Added

• gated verification class — the original principle in one construct: a programmatic (or any) gate runs first;
  failing it scores 0 and never calls the judge; only passing answers are scored by the score verification
  (typically a rubric). gate_min sets the pass line for continuous gates.
• judge_check / bench_tasks: judge-scored tasks (rubric / gated / trajectory / retrieval) now flow through the
  same paired pipeline (run_interleaved → report), binarised at an explicit threshold.
• Project website at https://alloevil.github.io/paired-eval/ (GitHub Pages from docs/): a single self-contained
  index.html generated by build_site.py — no external scripts, styles or fonts; zh/en toggle; the demo output,
  version and install command are pulled from the code, and tests/test_site.py fails if the file drifts from the generator.
• Logo (docs/assets/logo.svg: two bars with confidence-interval whiskers) and a centered README header.

Changed

• README and website reframed around the original question — evaluating models, agents and harnesses, verifying
  programmatically first and with rubrics only for what cannot be verified — instead of "compare two systems".
  The offline demo is now labelled as stubs standing in for models; the real-model quick start shows one A/B per
  object (model / harness / agent) on a task set that mixes exact, gated and trajectory classes.
• README restructured after comparing nine reference projects (hypothesis, httpx, ruff, ragas, deepeval, openai/evals, …):
  install and a real example with output within the first 20 lines, no argumentative table; one-line GitHub
  description (203 → 115 chars, no jargon); homepage set.

---

Verified on this tag: 210 tests on Python 3.9 / 3.10 / 3.13 locally and 3.9 / 3.12 / 3.13 in CI; changed-line mutation ratchet 27/27 killed for the new verification code; every README code block executed against the real API; website generated from code and asserted equal to the committed file.

paired-eval 0.2.0

paired-eval 0.2.0

Added

• paired_eval module: a single import surface matching the repository name (import paired_eval as pe), re-exporting
  the public API of the implementation modules, plus pe.demo() / python3 paired_eval.py — an offline demo that
  prints a complete report from two stub systems in under a second.
• claim_eval.fmt_p: report p-values below 1e-4 in scientific notation instead of 0.0000.
• Report text in English: set_language("en") (or lang="en" on interpret / report, --lang en on the demo).
  The Chinese strings are byte-identical to before; a test asserts the English output contains no CJK
  characters and that both languages expose the same placeholders.

Changed

• README rewritten from the user's perspective (why / install / ten-second demo / your own tasks / how to read the
  report / API / scope). The demo output pasted in both READMEs is asserted equal to the actual output by a test, and
  every code block is executed against the real API.
• README gains a flow diagram and a grounded "how it relates to other tools" section (lm-evaluation-harness,
  Inspect, promptfoo, openai/evals — descriptions taken from their own READMEs).

---

Verified on this tag: full fast suite on Python 3.9 / 3.10 / 3.12 / 3.13 (locally and in CI), hook integration, power calibration, hand-picked mutants; changed-line mutation ratchet 4/4 killed.

paired-eval 0.1.0

paired-eval 0.1.0 — first public cut.

First public cut. Everything below was built and verified in one continuous session; the reasoning behind each piece
is in docs/lessons.md, the measured results in docs/findings.md, and the
conclusions that were later overturned in docs/corrections.md.

Added

• Verifier spectrum: answer_match (exact / numeric / choice / set / \boxed{}), claim_eval (claim extraction,
  retrieval and trajectory grounding with three-valued verdicts), rubric_eval (per-criterion binary rubric judging
  with a canary against bluffing answers), eval_task (declarative task schema routed by verification class).
• Paired benchmarking (paired_bench): interleaved repeated runs, reliability matrix, pairwise comparison with
  Holm correction, saturation diagnostics, two-stage task screening (screen_tasks, screen_graded), and a single
  report() that emits nine diagnostics at once — effect size + CI, per-task permutation p, per-round McNemar, Holm,
  discordant-pair concentration, informative sample, refusals, the null's rule-out bound, ceiling/floor flags.
• Statistical primitives (claim_eval): paired_compare (with effective pairs), mcnemar_exact, holm_adjust,
  wilson_ci, pass_hat_k, required_tasks, required_pairs, detectable_effect, p_floor,
  min_units_for_alpha, interpret — the last translates a comparison into one of four honest verdicts
  (significant / bounded null / uninformative / powerless), naming the limiting basis and the remedy.
• Finding reproduction (reproduce_findings.py): three recorded conclusions as executable checks with
  drift-vs-power separation.
• Self-verification: 192 tests (~3 s), real-hook integration check, power calibration, 42 hand-picked semantic
  mutants, AST mutation ratchet over 473 points with a 30-entry archived equivalent-mutant baseline, crash-safe
  (SIGKILL) source backups, structural self-checks (defined == executed tests, doc anchors resolve, empty suite fails).
• Project shell: bilingual README whose examples are executed by tests, MIT license, pyproject.toml (zero
  dependencies, Python ≥ 3.9), CI matrix 3.9 / 3.12 / 3.13 plus a nightly full checklist, a standard-library-only
  OpenAI-compatible adapter example tested against a local HTTP server.

Fixed (before release, recorded for the method)

• interpret() computed the p floor from the unit count; McNemar's attainable minimum p depends on discordant
  pairs, and the permutation test's on nonzero-difference pairs. Found by the integration test on its first run.
• A boundary test used 0.88 − 0.80 as "exactly the threshold"; it is 0.0799… and passed on Python 3.10 only by
  rounding luck. Exposed by CI on 3.12 (Neumaier sum()); fixed with binary-exact values and a multi-interpreter layer.
• check_hooks.py pushed a hardcoded main to its temporary bare remote, so the hook check failed on any
  non-main branch or PR merge ref (Dependabot's first PR hit it). The temporary clone now always works on a
  fixed branch name, whatever is checked out.

---

Verified on this tag: 192 tests on Python 3.9 / 3.12 / 3.13, real-hook integration, power calibration, 42 hand-picked mutants, and the full AST mutation ratchet (446/473 killed, 27 survivors = archived baseline). Readers of docs/findings.md: the numbers there are instance-specific measurements, not results to inherit — see the boundary statement at the top of that file.