Per-mode (tau, beta_core) re-tuning sensitivity grid

[nikolay-e]

Context

Paper v2 runs all scoring modes at EGO-calibrated (tau=0.12, beta_core=0.5) — a standing reviewer objection to the mode ablation. Recorded as a sensitivity question; close it.

Tasks

• 5x3 grid (tau x beta_core) per mode {BM25, PPR, RRF, learned} on the calibration manifest; validation on held-out
• Report per-mode optima + sensitivity of the mode ranking to per-mode tuning

Acceptance

• table in paper v3; conclusion stated either way ("ranking robust to re-tuning" or updated ranking)

[nikolay-e]

Full v2-protocol sweep on the post-v5 graph is complete and archived — the raw material for this issue and its siblings now exists:

• results/final/v6full/ — 13 cells, each with PAPER_TABLE.md + per-benchmark JSON + budget-sharded checkpoints: ego_grid_new, ego_grid_oldpure (budget grids -1..128k on the flipped vs pre-cycle operating point), ppr_grid, bm25int_grid, bm25ext_grid, ablations ego_eps0/ego_boltzmann/ego_nobonus/ego_tau0, baselines patch_files_8k/random_8k/aider_fair_8k/aider_oracle_8k. Test splits 500x3, 600s/instance.
• results/final/v6/ — the six decision arms (admission x tau x ceiling factorial) + paired dcbench-372 pair; per-arm envs are pinned explicitly in results/final/v6_arms/*.json.
• results/stageA/ — 12-cell screening grid (admission x tau x mode) with corpus + dcbench per cell; results/calibration/v5_adm1{,_validation} — the tau x cbf surface + held-out.

Headlines at b8000/contextbench: ego(new) 0.848 recall / 0.224 precision vs aider repo-map fair 0.463 / oracle 0.490, patch_files 0.491, bm25ext 0.484 — ~1.8x recall over every external baseline at equal budget. Mode complementarity is sharp: ppr 0.368 precision, bm25int 0.869 recall at -28% tokens — the per-candidate-feature motivation for #129.

Issues this feeds: #140 (this one — per-mode re-tuning grids: stage A/B + the four mode grids), #143 (bm25 sensitivity: bm25int/bm25ext grids), #144 (line-level metrics live in the checkpoints), #169 (residual cells: eps0/boltzmann/nobonus/tau0 done; rho/alpha blend and edge-cap K remain), #129 (complementarity evidence). #141 (hub-suppression on/off) was NOT run — still open work.

[nikolay-e]

Premise correction, plus the grid's cheap half measured.

The context block says the modes run at "EGO-calibrated (tau=0.12,
beta_core=0.5)". That has not been the shipped point since the v5 flip
(#211, 2026-08-19): DEFAULT_STOPPING_THRESHOLD = 0.05,
DEFAULT_CORE_BUDGET_FRACTION = 0.4, plus the per-file admission gate. The mode
list {BM25, PPR, RRF, learned} also omits pit, which is the fusion arm that
has ever beaten EGO on a recall metric. Re-running the grid around the paper's
point would answer a question nobody is asking any more.

Measured today on the shipped configuration — all five modes, full corpus
(2902 cases), DIFFCTX_PROBE_MODE × DIFFCTX_YAML_IGNORE_BASELINE=1, one
session, one build:

mode	below-threshold	Δ vs ego	paper-era (tau=0.12)	Δ then
ego	68	—	371	—
pit	72	+4	410	+39
ppr	80	+12	—	—
rrf	80	+12	450	+79
bm25	157	+89	124	−247

(ego's 68 reproduces known_below_threshold.txt exactly — 68 entries in an
88-line file — so the baseline and the probe agree.)

Two things this changes.

BM25 inverted. It was 247 cases better than EGO and is now 89 worse. The old
number was the saturating-metric artifact QA.md long suspected: recall × (1 − forbidden_rate) does not penalise under-selection, and BM25 was conservative.
The admission gate removed that free ride. Any BM25 figure from the paper era —
including "ego ≈ bm25" on dcbench — was taken before the gate and does not carry
forward.

The fusion gap nearly closed. pit went from +39 to +4 of 2902, rrf from
+79 to +12. Fusion was not measured and found wanting; it was measured at a
calibration point it was never tuned for, and almost the whole gap was the
operating point rather than the fusion.

The result that actually matters is not the ranking — it is that the modes fail
different cases.

pair	shared failures	A-only	B-only
ego vs pit	52	16	20
ppr vs rrf	55	25	25

ppr and rrf both landing on 80 is a coincidence of counts, not a wiring bug —
their failure sets differ by 25 cases each way.

Intersecting all four graph-based modes:

ego ∩ pit ∩ ppr ∩ rrf  =  47 cases
best single mode (ego) =  68 cases

A perfect per-instance mode chooser would fail 47 instead of 68 — a 31%
reduction, from algorithms that already ship. Of EGO's 68 failures, pit
rescues 16, rrf 16, ppr 13.

Proposed re-scope of this issue. The 5×3 (tau × beta_core) grid per mode is
the expensive half and answers a reviewer objection. The cheap half — does the
mode ranking survive the current operating point — is answered above and says
yes for the ranking and no for the premise. What the numbers argue for next is
not re-tuning each mode in isolation but measuring the selectable envelope: an
oracle-chooser upper bound is now a concrete 47, and that is the number #129's
learned ranker has to be judged against, not a hand-set weight vector.

Caveat, stated so it is not read past: this is the corpus metric, which
saturates and does not penalise over-selection, and the per-case verdict is
binary. It is the right instrument for mechanism — deterministic, full
population, one build — and the wrong one for value. The recall-metric
comparison (dcbench/realworld at B=8000, paired, n≥100) is still owed, and its
last run predates the v5 default.

[nikolay-e]

Superseded by #246: under calibrated fusion every mode's scale is mapped to probabilities, so the per-mode (τ, β_core) re-tuning question becomes the calibration ablation (isotonic vs Platt vs none) in #246's appendix cells. v3 measures only the shipped v5 config (#252).