fix: make ctx smart deterministic and semantic-first#18
Merged
Conversation
`ctx smart <task>` selected the wrong files and was non-deterministic. For
"add a new output format to ctx sql" it returned generic hubs (main.rs,
smart_cmd.rs, analysis.rs) and unrelated test files while omitting
src/commands/sql.rs — even though that file's symbols are the top
`ctx semantic` hits — and the selected set changed on every run.
Two defects in rank_files (src/smart.rs):
1. Scale inversion. A direct SemanticMatch is scored with the raw
similarity, which the 1/(1+L2) conversion compresses to ~0.4-0.6, while
every depth-1 call-graph neighbor gets a flat 0.8 (CalledBy) / 0.7
(Calls). So a file pulled in purely by call-graph expansion outranked the
on-topic seed that produced it, then consumed the token budget first
(select_by_token_budget is greedy first-fit) and evicted the seed.
2. Non-determinism. rank_files sorted by relevance_score alone with no
tie-break, over a Vec collected from a HashMap. The many files tied at
0.8/0.7/0.5 came out in hash order, so which tied files survived the
budget varied per run.
Rework rank_files into a tiered, total ordering:
(1) files with a direct semantic match rank above graph-only files
(via new FileSelection::best_semantic_score), so call-graph expansion
supplements the seeds instead of displacing them;
(2) within a tier, higher score first (semantic score for the semantic
tier, relevance_score for the graph-only tier — never compared across
scales);
(3) ties break by path ascending, making selection deterministic.
Seeding, call-graph expansion, weight() constants, the add_reason max-merge,
and the token budget are all unchanged — tiering supersedes the cross-scale
comparison, so this is a ranking-only change.
Add unit tests for the tier ordering, intra-tier score/path ordering, and
determinism (same set in any input order yields the same ranked paths). The
existing tests only used single-tier Explicit reasons, so they never
exercised the semantic-vs-graph inversion.
Verified end-to-end: the query above now deterministically selects
src/commands/sql.rs at rank 1 across repeated runs.
saldestechnology
added a commit
that referenced
this pull request
Jul 11, 2026
Two residual `ctx smart` misses remained after #18 (which made selection deterministic and semantic-first), each with a distinct cause: - A candidate whose path literally contains a task word could still be outranked by the semantic matches and squeezed out by the token budget (e.g. `embeddings/openai.rs` for "…openai" — present only via a call-graph edge). A lexical signal fixes this. - The single most-relevant file could exceed the whole budget and be dropped by greedy first-fit, which then backfilled with smaller, less-relevant files (e.g. `parser/solidity.rs`, 9074 tokens > the 8000 budget, was already rank 1 yet vanished). A budget-policy fix is needed. Part A — lexical path boost: - New shared `utils::lexical_tokens`: lowercases, splits on non-alphanumeric and camelCase boundaries, drops length-≤-1 tokens and a small stopword set. - `FileSelection` gains `lexical_score` = count of distinct task tokens that appear in the file's PATH (path only, not symbol names: matching identifiers would let ubiquitous names like `ctx` — a test helper across `tests/*_cli.rs` — score a hit on nearly every file and drown out the on-topic one). - `rank_files` treats a lexical hit as top-tier alongside semantic matches and orders lexical hits first within the tier — surfacing the on-topic file without any tunable float weight. Determinism (path tie-break) is preserved. Part B — budget guarantee: - `select_with_guaranteed_top` always includes the rank-1 file even when it alone exceeds the budget, then first-fits the remainder (reusing `select_by_token_budget`). The handler prints a one-line stderr note when the top file is oversized. Adds unit tests for the tokenizer, lexical promotion/ordering, path-only scoring (asserting `ctx` in a symbol name does not match), and the oversized- top budget policy. #18's determinism/tiering tests still pass with `lexical_score = 0`. Verified end-to-end: "…openai" now selects `embeddings/openai.rs` at rank 1; "parse solidity contracts" includes `parser/solidity.rs` with the oversize note; "add a new output format to ctx sql" keeps `src/commands/sql.rs` at rank 1 and stays deterministic across runs.
saldestechnology
added a commit
that referenced
this pull request
Jul 11, 2026
…rt (#25) * feat: lexical path boost + oversized-top budget guarantee for ctx smart Two residual `ctx smart` misses remained after #18 (which made selection deterministic and semantic-first), each with a distinct cause: - A candidate whose path literally contains a task word could still be outranked by the semantic matches and squeezed out by the token budget (e.g. `embeddings/openai.rs` for "…openai" — present only via a call-graph edge). A lexical signal fixes this. - The single most-relevant file could exceed the whole budget and be dropped by greedy first-fit, which then backfilled with smaller, less-relevant files (e.g. `parser/solidity.rs`, 9074 tokens > the 8000 budget, was already rank 1 yet vanished). A budget-policy fix is needed. Part A — lexical path boost: - New shared `utils::lexical_tokens`: lowercases, splits on non-alphanumeric and camelCase boundaries, drops length-≤-1 tokens and a small stopword set. - `FileSelection` gains `lexical_score` = count of distinct task tokens that appear in the file's PATH (path only, not symbol names: matching identifiers would let ubiquitous names like `ctx` — a test helper across `tests/*_cli.rs` — score a hit on nearly every file and drown out the on-topic one). - `rank_files` treats a lexical hit as top-tier alongside semantic matches and orders lexical hits first within the tier — surfacing the on-topic file without any tunable float weight. Determinism (path tie-break) is preserved. Part B — budget guarantee: - `select_with_guaranteed_top` always includes the rank-1 file even when it alone exceeds the budget, then first-fits the remainder (reusing `select_by_token_budget`). The handler prints a one-line stderr note when the top file is oversized. Adds unit tests for the tokenizer, lexical promotion/ordering, path-only scoring (asserting `ctx` in a symbol name does not match), and the oversized- top budget policy. #18's determinism/tiering tests still pass with `lexical_score = 0`. Verified end-to-end: "…openai" now selects `embeddings/openai.rs` at rank 1; "parse solidity contracts" includes `parser/solidity.rs` with the oversize note; "add a new output format to ctx sql" keeps `src/commands/sql.rs` at rank 1 and stays deterministic across runs. * test: deterministic offline regression guard for ctx smart ranking Adds tests/smart_relevance.rs — a library-level integration test that guards the smart file-selection ranking against regressions without needing the fastembed model (which downloads ~90MB and is not byte-stable across arches, so it cannot run in CI). It hand-builds a tiny on-disk index (files + symbols + edges) and seeds exact embedding vectors via the library API (Database::store_embedding), then calls smart_context_with_embedding directly with a crafted query vector — exercising the full seed -> call-graph expand -> lexical -> rank -> budget pipeline deterministically and offline. Two scenarios: - path_match_beats_symbol_name_noise: a task token in a file's PATH must promote the on-topic file over a higher-semantic-scored file whose only match is a symbol NAME. This reproduces the `ctx` test-helper regression that was originally caught by hand. Verified as a real guard via mutation: reverting compute_lexical_score to also match symbol names makes this test fail. - graph_only_path_match_is_promoted: a candidate reachable only through call-graph expansion whose path matches the task is surfaced to the top rather than being outranked by the semantic matches and dropped. * fix: MCP analysis config variable shadowed SmartConfig under --all-features The project-config load added in #28 was named `config`, shadowing the `SmartConfig config` above it, so `smart_context_with_embedding(..., config)` passed the wrong type. Only compiled under the `mcp` feature, so the default build (and macOS CI) missed it; `--all-features` failed to compile. Rename the project config to `project_config`. (Latent on main from #28; this fix lands there via this PR.)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
ctx smart "<task>"selected the wrong files and was non-deterministic. For"add a new output format to ctx sql"it returned generic hubs (main.rs,smart_cmd.rs,analysis.rs) and even unrelated test files while omittingsrc/commands/sql.rs— despite that file's symbols (run_sql,render) being the topctx semantichits. Back-to-back runs of the same query produced different file sets (sql.rsappeared in only 2 of 5 runs).Root cause
Two defects in
rank_files(src/smart.rs):SemanticMatchis scored with the raw similarity, which1/(1+L2)(src/embeddings/mod.rs) compresses to ~0.4–0.6, while every depth-1 call-graph neighbor gets a flat 0.8 (CalledBy) / 0.7 (Calls). So a file pulled in purely by call-graph expansion outranked the on-topic seed that produced it, then consumed the 8000-token budget first (select_by_token_budgetis greedy first-fit) and evicted the seed.rank_filessorted byrelevance_scorealone with no tie-break, over aVeccollected from aHashMap. The many files tied at 0.8/0.7/0.5 came out in hash order, so which tied files survived the budget varied per run.(The underscore/
ESCAPEbug from #15 is unrelated —smartresolves the graph via id-basedcall_graph/impact_analysis, neverfind_symbols_filtered.)Fix
Rework
rank_filesinto a tiered, total ordering:FileSelection::best_semantic_score), so call-graph expansion supplements the seeds instead of displacing them.relevance_scorefor the graph-only tier; the two scales are never compared against each other.pathascending → deterministic across runs.Seeding, call-graph expansion,
weight()constants, theadd_reasonmax-merge, and the token budget are unchanged — tiering supersedes the cross-scale comparison, so this is a ranking-only change. (A future overhaul could normalize the semantic/graph scales and add a lexical path boost; noted, not done here.)Tests
The existing ranking tests only used single-tier
Explicitreasons, so they never exercised the semantic-vs-graph inversion. Added:test_semantic_match_ranks_above_graph_only— aSemanticMatch{0.5}file must outrank aCalledBy{depth:1}(0.8) file.test_semantic_tier_orders_by_score_then_path— intra-tier score order, path tie-break.test_rank_files_is_deterministic— same set in any input order → identical ranked paths.Verification
cargo fmt --checkandcargo clippy --all-targets -D warningsclean.ctx smart "add a new output format to ctx sql"run 5× now returns the identical set every time, withsrc/commands/sql.rsat rank 1. Spot-checks for "parse solidity contracts" and "generate embeddings with openai" surface the relevant implementation files (parser/mod.rs,embeddings/mod.rs) and are likewise deterministic.