Learned fusion (LTR) over per-candidate features — make-or-break

[nikolay-e]

Context

Make-or-break for the "graph has independent value" thesis. Deployed EGO mixture (hand-set weights) scores below its own lexical component (0.352 < 0.402); union proves the ceiling ≥0.473. Instead of hand-tuning 10 category multipliers, learn a linear ranker over per-candidate features. Runtime stays a dot product: deterministic, zero-model-cost identity preserved, weights are interpretable (and are the per-category calibration Section 4.3 promised).

Features (initial set, ~15–20)

bm25_score, ego_score, ppr_score, cochange_count, import_hop_distance, same_dir, test_link, path_token_overlap, identifier_overlap, fragment_kind one-hot, I(f) importance prior, file_size_tokens, in_degree, sibling, proximity_decay

Tasks

• Feature extraction pass emitting per-candidate vectors into telemetry (behind env flag)
• Train logistic regression + LambdaMART on co-change corpus (04) train split + ContextBench train; select on val
• Bake winning weights into Rust as ScoringMode::Learned; weights table committed + printed by --explain
• Full sweep at B=8000, seeds 42/43/44, paired stats vs RRF (01) and vs union artifact
• Ablate feature groups (graph-only, lexical-only, history-only) for the paper

Gate (pre-registered)

• held-out nontrivial_file_recall ≥ 0.50 AND file_precision ≥ 0.30
• If failed: RRF stays deployed default and paper v3 states plainly that the graph is a complement, not a dominant signal. Commit the negative result with REVERT-tag discipline.

Depends on: 04-cochange-corpus, 06-multihop-imports

[nikolay-e]

The ~500-gold-instance label ceiling breaks via #128 masked co-change: unlimited PR-derived training labels for the fusion learner; gold manifests reserved for eval only. HPO caveat: small validation sets show ~1.8% average meta-overfitting under adaptive search — keep the learner low-capacity (linear / RF over per-candidate features), few hyperparameters, nested CV with reshuffling.

This also sharpens the kill-criterion: if LTR trained on unlimited co-change labels still cannot beat calibrated PIT-fusion (#125 follow-up), the graph-has-independent-value thesis fails far more credibly than on 500 instances.

[nikolay-e]

Motivation upgraded from hypothesis to measurement by the v5/v6 cycle:

• The FIXED-formula fusion (pit) is dead: <= ego in all 12 screening cells on the corrected dcbench pool (n=145) and on the corpus; RRF fusion scoring mode (candidate new default) #125's flip criterion unmet everywhere. The 2026-08-04 night result (+13%) did not survive the dataset fixes (dcbench annotation hygiene: re-annotate nontrivial gold for the 12 corrected instances + protocol conformance sweep #195) and the un-hanging of sentry/gitpod (near-dense edge emission on huge same-directory trees: 199M raw edges / 37GB peak on one mui instance #116).
• Component complementarity is sharp and mechanistic (test splits, b8000, n=500/bench, results/final/v6full): ppr file_precision 0.368 vs ego 0.224 at -2.5pt recall; bm25int recall 0.869 (> ego 0.848) at -28% tokens; ego uniquely strong on dcbench nontrivial gold. Orthogonal strengths = exactly the per-candidate feature signal LTR consumes and a fixed blend cannot.

Plan: this issue is the next MAJOR cycle, gated on #128 (training corpus >= 5k instances) — LTR without a corpus is how pit died. Order: #128 mining pipeline -> feature extraction over the three component scores + graph/lexical per-candidate features -> pre-registered kill criterion vs the v5 default (paired dcbench nt + corpus + test splits, same bars as #65). Interim cheap win shipped separately: --scoring presets documented per use case (ego = review default, bm25 = cheap high-recall candidate supply, ppr = precision-first).

[nikolay-e]

Pre-registered gate correction — the failure branch names a world that never existed.

The gate reads:

If failed: RRF stays deployed default and paper v3 states plainly that the graph is a complement, not a dominant signal.

RRF has never been the deployed default. config/limits.rs:49 has read
DEFAULT_SCORING = "ego" throughout; #125 ("RRF fusion scoring mode (candidate
new default)") was closed 2026-08-04 without a flip, on the measurement that
RRF scored 450 below-threshold corpus cases against EGO's 371. Recording this
here rather than editing the gate silently: a pre-registered criterion loses its
force the moment it is reinterpreted after the fact, so the substitution is
stated, not performed.

Read the failure branch as: EGO stays the deployed default. The conclusion
it licenses is unchanged.

Two further corrections to the context block, both from live code:

• The premise "deployed EGO mixture scores below its own lexical component
  (0.352 < 0.402)" comes from the paper-v2 calibration point. The shipped
  configuration has moved twice since (v5, 2026-08-19): tau 0.12 → 0.05
  (DEFAULT_STOPPING_THRESHOLD), core_budget_fraction 0.5 → 0.4, plus the
  per-file admission gate. Any "make-or-break" comparison must be against that
  point, not against the paper's.
• The feature list should carry the fusion components that now exist —
  pit_score alongside bm25_score/ego_score/ppr_score. PIT is the one
  fusion arm that has ever beaten EGO on a recall metric (dcbench, 2026-08-04),
  and excluding it makes the learned ranker unable to express an arm already
  known to be competitive.

The dependency is the real blocker, and it should be stated as a gate rather
than a footnote. "Depends on: 04-cochange-corpus" (#128) is what decides
whether this issue is answerable at all: every mode comparison run so far sits
at n≈102 with a confidence interval containing zero. A learned ranker trained
and selected on that sample size cannot clear a ≥0.50 recall bar with evidence
anyone should act on — it can only overfit it. #128 first, then this.

[nikolay-e]

The make-or-break question now has an upper bound, measured today.

Full detail and method in #140. The part that bears directly on this issue:

On the shipped configuration, over the full 2902-case corpus, the four
graph-based modes fail substantially different cases.

ego                    68 below-threshold   (best single mode)
pit                    72                   (rescues 16 of ego's 68)
rrf                    80                   (rescues 16 of ego's 68)
ppr                    80                   (rescues 13 of ego's 68)

ego ∩ pit ∩ ppr ∩ rrf  47                   (perfect per-instance chooser)

47 is this issue's ceiling for the existing feature set — a 31% reduction over
the best single mode, available with no new signal at all. That reframes the
gate. The current one asks whether a learned ranker clears absolute
nontrivial_file_recall ≥ 0.50 / precision ≥ 0.30; what it should also ask is
what fraction of the 68 → 47 gap it recovers, because that fraction is the only
part attributable to learning rather than to the features already present.

It also suggests a cheaper first rung than LambdaMART over ~20 features: a
per-instance mode selector — a classifier over instance-level properties
(candidate-universe size, changed-file count, language, fragment-kind mix,
lexical/graph score agreement) picking among four scorers that already ship.
Runtime cost is one extra scoring pass or a cheap predicted choice; determinism
and zero-model-cost survive; the weights stay readable. If a selector cannot
recover a meaningful share of 21 cases, a 20-feature ranker over the same signal
is unlikely to, and that is a cheap kill-criterion to reach.

Unchanged: #128 is still the blocker. 21 cases and an n≈102 recall benchmark
cannot select a model without overfitting the selection set. Label supply first.

Method note so this is reproducible rather than quoted: five runs of
DIFFCTX_PROBE_MODE=<mode> DIFFCTX_YAML_IGNORE_BASELINE=1 cargo test --profile release-unwind --test yaml_cases, one build, one session; failure
lists diffed with comm. EGO's set reproduces known_below_threshold.txt
exactly, which is the check that the probe and the shipped gate agree.