v0.5.0 — ContextBench downstream evidence, and a public-claims audit

First release since v0.4.1 (2026-06-14). Two things are new: a downstream result, and a correction pass over every number this project publishes.

Downstream evidence

Context retrieval was previously justified only by retrieval metrics. It is now measured end to end.

On 128 ContextBench Python tasks, judged by each repository's own test suite (no LLM-as-judge), context roughly quadruples pass@1: 5.5% → 25.8%, exact McNemar p < 0.0001 on every context arm against the no-context baseline.

The honest companion, reported alongside: the three context variants (default / gap / depboost) are statistically indistinguishable from each other (p = 0.36 / 0.81 / 0.65). Only ~19 of 128 tasks are discordant on any variant pair, so the benchmark is underpowered to resolve a 3pp difference — and more seeds would not fix that, since the limit is discordant-pair count. The win is context versus no context, not this selector versus that one.

Full results and method: benchmarks/contextbench/RESULTS.md. Reusable Wilson-interval and exact-McNemar helpers: benchmarks/contextbench/stats.py.

Public claims corrected

An audit of the README, docs, and hosted site against the underlying artifacts found six claims that had drifted:

• "~2× the recall of grep at every token budget" → 1.5–2.7× depending on budget (the flat 2× did not hold at 1k, where it is +47%).
• Sample size stated as 423 commits / 7 repos in some places and 701 / 9 in others → unified.
• --cutoff gap was stated flatly as 4× precision / ~30% recall; that is the co-change benchmark. Now scoped, with the ContextBench figures (2.2× / ~14%) alongside.
• The calibration example on the docs site still showed r=0.274 (n=25) — the number the 2026-07 rigor pass retracted as measured on a polluted index. Replaced with the clean r=0.287 at n=1,080, and the retraction is stated inline.
• The site described downstream LLM evaluation as "not yet answered." It is answered above.
• docs/RERANK.md described a reranker as shipped. It is not in the package — it lives on the eval/rigor-pass branch, and where it was tested downstream it did not help. Now carries a NOT SHIPPED banner.

Housekeeping

• __version__ was stuck at 0.3.0 while releases claimed 0.4.x. Now correct.
• The PyPI publish workflow required a trusted publisher that was never configured, so it failed on every version tag. Removed; the package installs from source and the README says so.
• README rewritten to lead with what the project is and who it is for, with an explicit "What this is not" scope section.
• lint-and-types CI failure fixed.

Known gap: the per-repo hit/recall table in the README does not currently regenerate from the committed benchmarks/results/eval_v2/ summaries, and those summaries do not record the blend weights used. Being re-run and re-committed; treat that table as provisional until it is.

DiffContext v0.4.1 - Class Ownership Extraction

Highlights

Added ownership-aware function extraction.

Before:

./core.py:invoke

After:

./core.py:Command.invoke
./core.py:Group.invoke
./core.py:Context.invoke

Benefits

• Eliminates same-file method collisions
• Preserves class ownership
• Improves graph precision on object-oriented repositories
• Supports async methods
• Enables accurate benchmarking on Click, Flask, FastAPI, and Rich

Validation

Click repository extraction:

Before: 371 functions
After: 506 functions

Next

v0.4.2 - Ownership-Aware Call Resolution

DiffContext v0.3.1

DiffContext v0.3.1

Highlights

This release adds repository-aware dependency resolution improvements and validates DiffContext on a real-world open-source repository.

Added

• Cross-file import resolution
• Attribute-call dependency resolution (self.method(), obj.method())
• Repository-wide dependency graph improvements

Benchmarks

Requests: api.request

• Total functions: 228
• Retrieved functions: 8
• Token reduction: 95.9%
• Runtime: ~220 ms

Requests: sessions.send

• Total functions: 228
• Retrieved functions: 47
• Token reduction: 71.6%
• Runtime: ~220 ms

Impact

DiffContext can now traverse significantly larger portions of object-oriented Python codebases and generate focused context subsets from real repositories.

Next Milestone

v0.4.0 — Real Repository Benchmarks

Target repositories:

• Click
• Flask
• Rich
• FastAPI

Goal:

• Validate dependency retrieval quality
• Measure token reduction across multiple OSS projects
• Identify remaining graph-recall gaps

DiffContext v0.2.0 - Benchmark Suite

Added

• Repository-wide context selection
• Benchmark framework
• Recall metric
• Precision metric
• Context reduction metric
• Runtime metric

Benchmark Results

simple_repo

Recall: 100%
Precision: 100%
Reduction: 0%

medium_repo

Recall: 33.33%
Precision: 100%
Reduction: 75%

Known Limitations

• Cross-file import resolution not implemented

Next

v0.3.0 Import Resolution

DiffContext v0.1.0

DiffContext v0.1.0

Initial public release of DiffContext, a static-analysis-based context compiler for LLM-assisted software engineering.

Features

• AST-based function extraction
• Repository state tracking
• Function-level diff detection
• Dependency graph generation
• Blast radius analysis
• Impact scoring
• Dependency expansion
• Context compilation

Pipeline

Repository
↓
AST Extraction
↓
State Tracking
↓
Diff Detection
↓
Dependency Graph
↓
Blast Radius Analysis
↓
Impact Scoring
↓
Dependency Expansion
↓
Context Builder
↓
Compiled Context
↓
LLM

Purpose

DiffContext transforms code changes into targeted LLM-ready context by identifying:

• What changed
• What depends on it
• What is affected
• What code should be included for reasoning

Current Scope

Implemented:

• Single-file Python repository analysis
• Function-level dependency tracking
• Blast radius propagation
• Context generation for changed code

Future Work

• Multi-file repository support
• Import resolution
• Class dependency analysis
• Context ranking improvements
• Repository-scale benchmarking