feat(search): discount the exact-name bonus by corpus frequency (#982)

[maxmilian]

Implements the corpus-frequency discount from #982 — the general lever #746 floated and left unbuilt.

The change

nameMatchBonus handed a flat 80 (whole query === name) / 60 (a token of a multi-word query === name) no matter how many symbols carry that name. STOP_WORDS deliberately doesn't filter common symbol names, so a bare usage / get / status collected the full bonus and outranked product code that answers the query but doesn't literally contain the token.

nameMatchIdfScale(df, total) = log(1 + total/df) / log(1 + total)

• a unique name yields exactly 1 — the common case is bit-for-bit unchanged
• weight decays as the name spreads across the corpus
• floored at 0.25, because explore/query relevance: a generic token's exact name-match overboosts in peripheral dirs — follow-up to #746 #982's own validation caveat asks for exactly this: a query where the generic token genuinely is the discriminating term (an actual usage() reporter) must still rank. This discounts the signal; it does not erase it.

Only the two exact-name arms are scaled. Prefix/substring bonuses are already small and length-scaled and never produced the crowd-out. Corpus stats reach the scorer through an optional parameter, so nameMatchBonus stays pure and every existing caller behaves exactly as before when stats aren't supplied.

Because findRelevantContext retrieves through searchNodes, the explore path inherits this. That closes the "0.3× of a huge number is still huge" gap #982 describes: the existing isDistinctiveIdentifier demotion now multiplies a base that is itself already discounted.

Measured effect

Fixture: #982's layout — product code with no symbol named usage, plus peripheral optional-skills/*/scripts/ helpers that each define a module-level usage() — scaled so the token is genuinely corpus-common (69 nodes, 24 named usage). Query: desktop status bar context window usage.

	top of the ranking
before	every top slot is an optional-skills/** usage(), all tied at 71.3; product code buried
after	the helpers leave the top 11 entirely; StatusBarController / StatusBar / ContextWindowMeter fill it

One thing this does NOT fix — worth being explicit

#982's 8-file minimal reproduction is not fixed by this change, and I'd rather say so than let the issue close on a half-claim.

In that repro only two symbols are named usage out of ~25 nodes. The token is rare there, so the IDF scale is ≈0.8 — nearly inert, which is correct behaviour for a corpus-frequency lever. I measured it: the helpers still land at 62.8 vs the top product symbol at 51.2.

So the repro demonstrates the mechanism (the exact-name bonus dominates) but not the condition this lever keys on (the name being common). What fixes that shape is the issue's complementary, deliberately-separate path lever — user-extensible de-prioritization via codegraph.json. I've kept it out of this PR since #982 explicitly asks for the two to stay distinct, and there's a test pinning the rare-name case so nobody later "fixes" the inertness by removing the floor.

Happy to follow up with the deprioritize lever as a second PR if you want it.

Tests

__tests__/name-match-idf.test.ts, 12 tests — all fail on main, pass here:

• nameMatchIdfScale: unique name → exactly 1; monotone decay; a very common name discounted but non-zero; degenerate inputs (df=0, total=1, NaN, df > total) safe
• nameMatchBonus: unchanged without corpus stats; unchanged for a rare name; discounted for a common one; prefix/substring arms untouched
• the end-to-end ranking flip above, plus a control that a query for usage still surfaces usage() in the top 3
• the explicit rare-name test documenting the limitation

Ranking-adjacent suites, all green: context-ranking, explore-corroboration-ranking, explore-nl-stopword-collision, explore-result-count, symbol-lookup, same-name-disambiguation, field-name-retrieval, search-query-parser — 61 passed. tsc --noEmit clean.

What I have not run — please tell me how you want this gated

CLAUDE.md requires a scorer change to be A/B'd with scripts/agent-eval/run-all.sh across the README repos plus a control, ≥2 runs/arm, before merge. I have not run that, so treat the numbers above as fixture-level evidence only, not the sufficiency proof the repo actually gates on. __tests__/evaluation/runner.ts isn't a substitute — its cases are pinned to another codebase's symbols (TransportService, …), so it can't be pointed at codegraph itself.

Two things worth your call before I burn the runs:

1. Is the floor right at 0.25? It's the one number I picked rather than derived, and it's exactly what the A/B would tune.
2. One COUNT(*) per search. nameCorpusStats() reads the total once per searchNodes call and memoizes per-name counts for that call. That's cheap relative to FTS, but it is a new query on the hot path — if you'd rather cache the total on the instance and invalidate on index write, say so and I'll switch it.

Draft until then.