v0.5.0 — ContextBench downstream evidence, and a public-claims audit

First release since v0.4.1 (2026-06-14). Two things are new: a downstream result, and a correction pass over every number this project publishes.

Downstream evidence

Context retrieval was previously justified only by retrieval metrics. It is now measured end to end.

On 128 ContextBench Python tasks, judged by each repository's own test suite (no LLM-as-judge), context roughly quadruples pass@1: 5.5% → 25.8%, exact McNemar p < 0.0001 on every context arm against the no-context baseline.

The honest companion, reported alongside: the three context variants (default / gap / depboost) are statistically indistinguishable from each other (p = 0.36 / 0.81 / 0.65). Only ~19 of 128 tasks are discordant on any variant pair, so the benchmark is underpowered to resolve a 3pp difference — and more seeds would not fix that, since the limit is discordant-pair count. The win is context versus no context, not this selector versus that one.

Full results and method: benchmarks/contextbench/RESULTS.md. Reusable Wilson-interval and exact-McNemar helpers: benchmarks/contextbench/stats.py.

Public claims corrected

An audit of the README, docs, and hosted site against the underlying artifacts found six claims that had drifted:

• "~2× the recall of grep at every token budget" → 1.5–2.7× depending on budget (the flat 2× did not hold at 1k, where it is +47%).
• Sample size stated as 423 commits / 7 repos in some places and 701 / 9 in others → unified.
• --cutoff gap was stated flatly as 4× precision / ~30% recall; that is the co-change benchmark. Now scoped, with the ContextBench figures (2.2× / ~14%) alongside.
• The calibration example on the docs site still showed r=0.274 (n=25) — the number the 2026-07 rigor pass retracted as measured on a polluted index. Replaced with the clean r=0.287 at n=1,080, and the retraction is stated inline.
• The site described downstream LLM evaluation as "not yet answered." It is answered above.
• docs/RERANK.md described a reranker as shipped. It is not in the package — it lives on the eval/rigor-pass branch, and where it was tested downstream it did not help. Now carries a NOT SHIPPED banner.

Housekeeping

• __version__ was stuck at 0.3.0 while releases claimed 0.4.x. Now correct.
• The PyPI publish workflow required a trusted publisher that was never configured, so it failed on every version tag. Removed; the package installs from source and the README says so.
• README rewritten to lead with what the project is and who it is for, with an explicit "What this is not" scope section.
• lint-and-types CI failure fixed.

Known gap: the per-repo hit/recall table in the README does not currently regenerate from the committed benchmarks/results/eval_v2/ summaries, and those summaries do not record the blend weights used. Being re-run and re-committed; treat that table as provisional until it is.