Repository navigation
Releases: technobasant/evalproof
Releases · technobasant/evalproof
Release list
evalproof 0.1.1
Fixed
- The Wilson interval now returns exactly 0 at k=0 and exactly 1 at k=n. Computing
the boundaries by subtraction left platform-dependent rounding (2.8e-17 on x86_64
Linux, 0.0 on arm64 macOS).
Changed
- The three-day
exclude-newerresolution guard is declared in[tool.uv], so
uv.lockmeans the same thing on every machine and in CI. - CI pins uv to the version that wrote
uv.lock.
evalproof 0.1.0
First release.
Added
- Suites in YAML: a JSONL golden set, a system under test, scorers, an optional judge,
calibration settings and a gate. - Systems under test: Ollama, Anthropic and OpenAI models with a prompt template; Python
functions; HTTP endpoints. System failures are recorded per case, never fatal. - Scorers: exact match, contains, regex with accept/reject answer keys, a JSON Schema
subset, numeric tolerance, recall@k, MRR, nDCG, and an LLM judge with a strict JSON
verdict, one parse retry and an explicit undecided state. - Pairwise judging with order swap; position-inconsistent verdicts become ties.
- Judge calibration: confusion matrix, TPR, FPR, Cohen's κ, and prevalence-corrected pass
rates by adjusted classify-and-count (bootstrap interval) and prediction-powered
inference (normal interval). - Paired comparison of two runs: exact McNemar for pass/fail, paired bootstrap for scores.
- A CI gate that fails only on a drop that is both beyond
max_dropand significant at
alpha, plus p95-latency and cost budgets, with distinct exit codes. - Content-addressed response cache; runs record the Ollama version and model digests.
- Markdown reports for GitHub step summaries, JSON reports, and a composite GitHub Action.
- Two examples: an offline ticket router (CI's gate contract) and a support assistant for
a fictional product judged by a local model, with measurements.