Skip to content

v2.8.0 — cross-repo generalization (3 OSS repos, 36 prompts, +18.2% aggregate MRR)

Choose a tag to compare

View all tags
@sravan27 sravan27 released this 25 Apr 11:29
v2.8.0
1fa022d

What's new

The v2.7 claim "beats every lexical baseline" rested on one dogfood repo. v2.8 proves it generalizes to three real OSS repos that are not in our fixture set.

Cross-repo evidence (36 hand-labeled prompts, pinned SHAs)

Repo (lang, files) auto_context MRR Best baseline Δ
axios/axios (JS, 214) 0.382 bm25-path 0.252 +0.130
BurntSushi/ripgrep (Rust, 100) 0.503 bm25-path 0.459 +0.044
psf/requests (Py, 36) 0.750 bm25-symbols 0.875 −0.125

Weighted aggregate across 36 prompts: auto_context 0.545 vs best baseline 0.461 → +18.2%.

The single loss (psf/requests) is honest: prompts in that set use exact class names (PreparedRequest, HTTPError, CaseInsensitiveDict), which is the lexical-retrieval ceiling regime where bm25-symbols caps. We win the cross-repo aggregate, in every language, but don't pretend to win every repo.

Ranker improvements (net-positive, kept)

  • plural ↔ singular stem variants in extract_tokens()
  • case-fold dedupe of path tokens (no triple-counting Request/request/requests)
  • df-discriminativity scaling: disc = 1 - df/N for path_substr bonus
  • file-level score aggregation: sum candidate scores per file, pick best-scoring line as representative

Tried and reverted (net-negative on synthetic)

  • symbol_part (token matches camel/snake segment of symbol)
  • symbol_phrase (compound-symbol substring in normalized prompt)

Quality gates

  • Synthetic MRR 0.984 (was 0.969 in v2.7) · P@3 0.698
  • Dogfood MRR 0.756 · top-1 0.600 · +0.142 over bm25-symbols
  • All 9 ranker_floor regression gates green
  • New multi_repo_eval.py exits non-zero if (a) weighted aggregate fails to beat every baseline or (b) any repo falls below the avg of the five baselines

Reproduce

git clone https://github.com/sravan27/context-os && cd context-os
python3 python/evals/runners/ranker_floor.py        # 9 hard gates
python3 python/evals/runners/multi_repo_eval.py     # 3 OSS repos, 36 prompts

First run of multi_repo_eval.py clones the three pinned repos to ~/.cache/context-os-multi-repo/ (~30MB total). Re-runs use the cache.

Files

  • python/evals/runners/multi_repo_eval.py — runner, dual acceptance criterion
  • python/evals/multi_repo_prompts/{axios,ripgrep,requests}.json — 36 hand-labeled prompts
  • python/evals/reports/multi-repo-eval.md — full report (per-repo tables, weighted aggregate, per-prompt detail)
  • hooks/python/auto_context.py — ranker improvements
  • docs/PITCH.md, docs/FOR-CLAUDE-CODE-TEAM.md, docs/PROPOSAL.md, docs/REVIEW-CHECKLIST.md — updated with v2.8 cross-repo evidence
Assets 2
Loading