explore/query relevance: a generic token's exact name-match overboosts in peripheral dirs — follow-up to #746

[michaelHMK]

Summary

Follow-up to #746 (and #720 / #745). Those landed the project-name token down-weighting. #746 also floated a more general idea:

discount a token that matches a large fraction of indexed paths/symbols (it carries ~no signal but currently dominates).

That corpus-frequency discount is still not implemented on main (1.1.0). As a result, a generic non-stopword query token that happens to exactly match a symbol name (e.g. a bare usage) wins a large nameMatchBonus and crowds out the on-target product code — both in codegraph query / codegraph_search (no dampening at all) and, to a lesser but still user-visible degree, in codegraph_explore (where an existing dampening fires but is insufficient).

What actually happens today (corrected root cause)

There are two scoring paths, and they behave differently — the original write-up conflated them:

1. codegraph query / codegraph_search (db/queries.ts searchNodes): score = bm25 + kindBonus + scorePathRelevance + nameMatchBonus. The only low-signal-token handling is dropping the project name (scorePathRelevance, via deriveProjectNameTokens). There is no corpus-frequency / generic-token discount, so a bare usage symbol ranks at the very top.

2. codegraph_explore (context/index.ts findRelevantContext): this path does already demote generic exact-name matches — isDistinctiveIdentifier(token) is false for a plain lowercase word like usage, so a common-word exact match with no 2nd corroborating term gets score *= 0.3 (and a generic single-term match gets *= 0.6). This demotion fires for the reported case — and is still not enough, because nameMatchBonus hands a bare exact-name match a score so high that 0.3× of it still beats product symbols that don't literally contain the token.

So the accurate statement is not "generic-token downweighting isn't done" — it's "the existing demotion is a binary heuristic (distinctive-vs-common + 2-term corroboration) that can't counter a large nameMatchBonus, and the plain query/search path has no such demotion at all." The #746 general corpus-frequency discount is what's missing.

The lever (nameMatchBonus, src/search/query-utils.ts)

For a query token that exactly equals a symbol name:

• whole query (whitespace-stripped) === name → 80
• in a multi-word query, a token === name → 60 ← this is the case here (the earlier draft said "80")

STOP_WORDS deliberately doesn't filter common symbol names (get/set/usage/update/status), so these tokens keep full weight even when they match thousands of symbols. matchesNonProductionDir's de-prioritized set is hardcoded (integration/sample/example/fixture/benchmark/demo) and not extensible, so a peripheral tree like optional-skills/.../scripts/ gets no de-prioritization.

Minimal reproduction (public, no private repo needed)

Layout — product code with no symbol literally named usage, plus two standalone helpers that each have a module-level def usage():

apps/desktop/statusbar/StatusBar.ts            class DesktopStatusBar { render, refresh, mount }
apps/desktop/statusbar/StatusBarController.ts  class StatusBarController
apps/desktop/context/ContextWindowMeter.ts     class ContextWindowMeter { read, recompute }, estimateTokens
apps/desktop/context/format.ts                 formatTokens
gateway/server/server.ts, packages/core/util/strings.ts   (filler)
optional-skills/bodyfat/scripts/body_calc.py   def usage(): print("usage: ...")
optional-skills/nutrition/scripts/macro_calc.py def usage(): print("usage: ...")

codegraph init + codegraph index, then query desktop status bar context window usage:

Run	Entry	Where the usage() helpers land
A. codegraph query (no dampening)	searchNodes	ranks 1 and 2 (≈7626% vs product's top ≈5104%)
B. codegraph explore (0.3× demotion active)	findRelevantContext	top two Source Code blocks — push StatusBarController.ts + format.ts out of the section
C. control: same query with usage→tokens	codegraph explore	helpers don't appear; 4 product files fill the section
D. causal probe: rename usage()→print_usage(), re-index	codegraph explore	helpers vanish entirely; product code reclaims every slot

D isolates the cause to a single signal: removing the exact token match (so the 60-point nameMatchBonus no longer applies) removes the symptom. The existing explore-path demotion was active the whole time and didn't prevent B.

Why the existing user-side levers don't help

• codegraph_explore / CLI explore accept only query, maxFiles, projectPath. projectPath selects which index — there's no sub-path / glob scope ("only apps/desktop", "don't rank optional-skills/").
• codegraph.json (ProjectConfig) supports only extensions and includeIgnored. No exclude; includeIgnored is the opposite lever.
• .gitignore doesn't apply — the peripheral tree is intentionally tracked content, so the Bug: Committed vendor/ directory is indexed in Go project despite .gitignore, resulting in bloated database (v0.9.2) #316 / v0.9.4 indexes node_modules in non-git projects, causing noisy context results #407 / [BUG] Currently, it is impossible to turn off the indexing of nested repositories #976 "don't index gitignored dirs" line doesn't cover it.

Proposal

Primary (the #746 follow-up — the part that's genuinely missing):

• Add a corpus-frequency / IDF-style discount so a query token matching a large fraction of indexed symbols/paths contributes proportionally less to nameMatchBonus (and scorePathRelevance). This lowers the base an exact-name match starts from, which the explore path's existing 0.3× demotion then multiplies — fixing the "0.3× of a huge number is still huge" gap, and also fixing the query/search path that has no demotion at all.

Complementary / independently shippable, lower risk — keep these two distinct (they have different semantics):

• Deprioritize / scope (a ranking lever): make the non-production-dir de-prioritization user-extensible via codegraph.json (e.g. deprioritize: ["optional-skills/", "scripts/"]), and/or add a --within <glob> / --exclude <glob> scope to explore / query + the MCP input schema.
• Exclude (an index/recall lever) is a bigger product decision — name it separately if pursued; it's not required to fix the ranking symptom above.

Validation caveat

This touches the scorer for every query, so per codegraph/CLAUDE.md and the note in #746 it must be A/B'd with scripts/agent-eval/run-all.sh across the README repos plus a control before merge. Edge case to preserve: a query where the "generic" token genuinely IS the discriminating term (an actual query about a usage() reporter) — IDF should discount, not erase, the signal.

[maxmilian]

Reproduced the root cause on current main — the analysis holds. For anyone triaging, the three grounding points:

• nameMatchBonus (src/search/query-utils.ts:343) hands a flat 80 (whole-query === name) / 60 (multi-word token === name), independent of how many symbols share that name.
• searchNodes (src/db/queries.ts:1024-1030) is purely additive: bm25 + kindBonus + scorePathRelevance + nameMatchBonus, with no corpus-frequency term.
• The exact-name supplement just above it (queries.ts:994-1020) re-injects any name = term node seeded at maxFtsScore, so a bare usage symbol enters rescoring at maxBM25 + kindBonus(function=10) + 60 — which is why it crowds out product code that doesn't literally contain the token. #746 closed with only the project-name down-weight landed; the general low-discriminative-token discount it floated is still absent.

One correction to the issue's premise (it's ~2 weeks old): codegraph.json does have exclude now — it landed in 45d3293 as a gitignore-style, index-level skip (src/extraction/index.ts:489). So the immediate symptom has a workaround today: "exclude": ["optional-skills/"] keeps those usage() helpers out of the index entirely. That's the recall lever the issue set aside, though — it removes the symbols rather than ranking them correctly.

So the genuinely-open, recall-preserving gap is still the ranking side, and it splits cleanly into two independently-shippable pieces:

1. Extensible deprioritize (low risk, ranking-only): matchesNonProductionDir (query-utils.ts:325) is a hardcoded set (integration/sample/example/fixture/benchmark/demo). Making it user-extensible via codegraph.json (e.g. deprioritize: ["optional-skills/", "scripts/"]) is a small, contained change with no scorer-wide blast radius — it only lowers scorePathRelevance for opted-in trees.
2. Corpus-frequency / IDF discount (the real Down-weight project-name / low-discriminative query tokens in search ranking (follow-up to #720) #746 follow-up, higher risk): scales nameMatchBonus/scorePathRelevance down by how large a fraction of indexed symbols a token matches. This touches every query, so per CLAUDE.md + the #746 note it needs an A/B via scripts/agent-eval across the README repos + a control that preserves the case where the "generic" token genuinely IS the discriminating term (a real query about a usage() reporter should discount, not erase).

@colbymchenry — happy to take (1) as a self-contained PR (extensible deprioritize, with the config validation mirroring the existing exclude/includeIgnored handling in project-config.ts and an eval A/B to show no regression), if that direction is welcome and if @michaelHMK isn't already on it. Would you rather see (1) first as the low-risk step, or go straight at the IDF discount in (2)?

[JerryPeace]

Another amplification surface for the same root cause, plus one negative data point that rules out a workaround.

The prompt-hook path makes this non-optional for the user

The existing analysis covers codegraph query / codegraph_search / codegraph_explore — paths where a human deliberately asked for a symbol. The UserPromptSubmit prompt-hook reuses the same ranking, but the "query" is the user's natural-language prompt. So a generic token that happens to be a common identifier triggers injection on prompts that reference no code at all, and the tokens are consumed silently on every turn with no user action to filter them.

Concrete case, v1.5.0, a ~785-file multi-language monorepo (Python / TypeScript / C#):

• Prompt: a meta discussion about indexing strategy and query relevance — zero references to any symbol, file, or feature in the repo.
• Injected: 15.3 KB of source plus call graph and blast radius for unrelated UI/query-layer symbols, selected purely because the prompt contained the words index and query.

The pattern is consistent for tokens that are simultaneously a domain concept and a common identifier: index, query, review, context, status. Those are exactly the words that dominate when the conversation is about the tooling or process rather than about product code — so the false-positive rate is highest in precisely the sessions where the injection has the least value.

Negative data point: scoping the index does not help

Before finding this issue I assumed the noise came from cross-project matches in the monorepo and that narrowing the index would fix it. Measured both ways, same query, same version:

	full index	one 179-file sub-project excluded
Files	785	606
Nodes	12,633	8,928
Edges	30,692	23,055
Hits for the generic token	10	10 (identical set)

Removing 23% of the corpus changed the hit set by zero, because the over-boosted matches were already first-party code that merely contained the token in a symbol name. So this is purely the ranking issue described above — not corpus composition, and not something users can work around by splitting or scoping their index.

Worth noting the excluded sub-project also cost ~21% of the genuine cross-directory edges between the two remaining projects (566 → 449), so index scoping is actively harmful as a mitigation here, independent of it not working.

Why the corpus-frequency discount from #746 would fix the injection case cleanly

A flat nameMatchBonus of 60/80 is indifferent to how many symbols share the name, which is the whole signal that matters for this class of token. For the prompt-hook path specifically, a corpus-frequency term would also give a natural way to inject nothing rather than something weak: if every candidate's score is dominated by a high-frequency exact-name match, that is a good indicator the prompt was not structural at all.

Happy to re-measure on a build with the discount if that helps validate it.