v2.6.0 — beats every lexical baseline on its own repo
TL;DR
auto_context — a ~400-line stdlib Python hook that injects file:line · symbol · imports candidates into the first Claude Code turn — now beats every lexical baseline on its own repo.
Live Claude A/B (unchanged from v2.5, re-validated)
- −40.9% aggregate tokens on 36 real
claude --printcalls, 95% bootstrap CI [32.7%, 48.9%] - 6/6 prompt-level wins, 16/18 per-run wins (Wilson CI [67.2%, 96.9%])
- Paired t-test p = 5.06e-07, Cohen's d = 1.84 (large effect)
- Wall-clock −35.3% (mean 11.80s → 7.64s)
Offline retrieval (Python/TS/Rust fixtures, 32 hand-labeled prompts)
- MRR 0.969 · P@3 0.703 · Coverage 1.000
- +0.094 MRR over BM25-symbols (textbook lexical baseline)
- +0.407 MRR over naive-filename
Dogfood on this repo (real heterogeneous codebase, 49 src, 440 symbols)
| Method | MRR | Top-1 | P@3 |
|---|---|---|---|
| auto_context | 0.800 | 0.667 | 0.322 |
| bm25-symbols | 0.619 | 0.533 | 0.244 |
| bm25-path | 0.536 | 0.467 | 0.256 |
| naive-filename | 0.483 | 0.400 | 0.322 |
| grep-count | 0.283 | 0.133 | 0.111 |
| random | 0.061 | 0.000 | 0.000 |
+0.181 MRR over BM25-symbols on real-repo prompts — not just synthetic fixtures.
Operational
- p99 hook latency: 23ms @ 100 files, 46ms @ 1k, 173ms @ 10k (5× under 1s SLA)
- 18/18 adversarial robustness cases pass (unicode, 100k prompts, null bytes, corrupt graph, regex bombs, shell meta, path injection)
- Ablation:
path_substr(ΔMRR −0.062) +path_exact(ΔMRR −0.016) load-bearing. No dead weight.
What changed vs v2.5
Ranker upgrades (hooks/python/auto_context.py):
- IDF-weighted symbol + path matches, dampened & capped at 1.6
- Basename-in-prompt detection across under/space/none normalizations (+15)
- Multi-token path coverage bonus (+2 per extra token, cap +8)
- NL → code term expansion (~20 curated mappings)
- Graph-aware stopwords: promote
test/file/find/... back to tokens when the graph has files named that way
CI: 5 new eval steps (ablation, robustness, latency, baseline-comparison, dogfood) now run on every PR.
Installer: setup.sh heredoc synced — installed users get the upgraded ranker immediately.
Install
curl -fsSL https://raw.githubusercontent.com/sravan27/context-os/main/setup.sh | bashAll evidence is reproducible: cd python/evals && ls runners/ → every number in this release has a Python script behind it.