Skip to content

v2.7.0 — Pitch-ready: ranker regression gate, 50k latency, ROI math

Choose a tag to compare

View all tags
@sravan27 sravan27 released this 24 Apr 16:22
v2.7.0
563a5b0

TL;DR

Acquisition-hardening release. Same ranker that got the −40.9% live-Claude result; harder evidence and a CI gate that prevents silent regression.

Three new docs for the Claude Code team (in order, ~30 min total):

What's new

Ranker regression gate (CI-enforced)

python/evals/runners/ranker_floor.py asserts 9 hard floors across synthetic / dogfood / baseline margins. Any PR that regresses retrieval quality red-lines the build.

Gate Floor Current
synthetic MRR 0.920 0.969
synthetic P@3 0.640 0.703
dogfood MRR 0.720 0.789
dogfood top-1 0.580 0.667
MRR lift over bm25-symbols (synth) +0.060 +0.094
MRR lift over naive-filename +0.300 +0.406
MRR lift over random +0.500 +0.562
dogfood MRR lift over bm25-symbols +0.080 +0.208

50k-file latency measurement

After path_df precomputation in the graph schema (v1→v2), the hook is O(tokens) per query, not O(files × tokens). Scale measurements:

Files p99 (v2.6) p99 (v2.7) SLA
100 25ms 23ms 1000ms
1,000 45ms 41ms 1000ms
10,000 175ms 118ms 1000ms
25,000 420ms 284ms 1000ms
50,000 830ms 589ms 1000ms (1.7× under)

ROI math

Conservative back-of-envelope (1M users × 400 prompts/month × 8.2K tokens saved × $6/1M tokens):

Without auto_context With Delta
Tokens/user/month 20M 11.8M −8.2M
Cost/user/month $120 $71 −$49
Across 1M users/year — — ~$588M

90% discounted for cache reuse / cohort overlap / power-user skew is still low-nine-figures/year.

Retrieval (held across the rebuild)

  • Synthetic MRR 0.969 · P@3 0.703 (unchanged)
  • Dogfood MRR 0.789 · top-1 0.667 (was 0.800; 49→50 files shifted path_df; honest number)
  • Beats every lexical baseline on real-repo prompts

CI

.github/workflows/ci.yml now runs ranker_floor.py as a hard gate after the dogfood step. Every PR regenerates every report; nothing ships without the floor holding.

No install action needed

v1 graphs are auto-detected; path_df is recomputed on first load. No rebuild required for existing users. curl -fsSL https://raw.githubusercontent.com/sravan27/context-os/main/setup.sh | bash pulls 2.7.0.

Reproduce everything in 5 minutes

git clone https://github.com/sravan27/context-os && cd context-os
python3 python/evals/runners/ranker_floor.py        # 9 hard gates, ~45s
python3 python/evals/runners/autocontext_eval.py    # MRR 0.969 · P@3 0.703
python3 python/evals/runners/dogfood_eval.py        # real-repo MRR 0.789
python3 python/evals/runners/robustness_test.py     # 18/18 adversarial
python3 python/evals/runners/latency_bench.py       # p99 SLA

Every number in the pitch is the output of one of those scripts. Nothing hand-edited. The green CI badge is the contract.

Contact

Email sridharsravan@icloud.com. We have code, time, and a strong bias for shipping. Happy to walk the upstream port with the Claude Code team.

Assets 2
Loading