Skip to content

fix: make ctx smart deterministic and semantic-first#18

Merged
saldestechnology merged 1 commit into
mainfrom
worktree-fix-smart-relevance
Jul 10, 2026
Merged

fix: make ctx smart deterministic and semantic-first#18
saldestechnology merged 1 commit into
mainfrom
worktree-fix-smart-relevance

Conversation

@saldestechnology

Copy link
Copy Markdown
Collaborator

Problem

ctx smart "<task>" selected the wrong files and was non-deterministic. For "add a new output format to ctx sql" it returned generic hubs (main.rs, smart_cmd.rs, analysis.rs) and even unrelated test files while omitting src/commands/sql.rs — despite that file's symbols (run_sql, render) being the top ctx semantic hits. Back-to-back runs of the same query produced different file sets (sql.rs appeared in only 2 of 5 runs).

Root cause

Two defects in rank_files (src/smart.rs):

  1. Scale inversion. A direct SemanticMatch is scored with the raw similarity, which 1/(1+L2) (src/embeddings/mod.rs) compresses to ~0.4–0.6, while every depth-1 call-graph neighbor gets a flat 0.8 (CalledBy) / 0.7 (Calls). So a file pulled in purely by call-graph expansion outranked the on-topic seed that produced it, then consumed the 8000-token budget first (select_by_token_budget is greedy first-fit) and evicted the seed.
  2. Non-determinism. rank_files sorted by relevance_score alone with no tie-break, over a Vec collected from a HashMap. The many files tied at 0.8/0.7/0.5 came out in hash order, so which tied files survived the budget varied per run.

(The underscore/ESCAPE bug from #15 is unrelated — smart resolves the graph via id-based call_graph/impact_analysis, never find_symbols_filtered.)

Fix

Rework rank_files into a tiered, total ordering:

  1. Files with a direct semantic match rank above graph-only files (new FileSelection::best_semantic_score), so call-graph expansion supplements the seeds instead of displacing them.
  2. Within a tier, higher score first — semantic score for the semantic tier, relevance_score for the graph-only tier; the two scales are never compared against each other.
  3. Ties break by path ascending → deterministic across runs.

Seeding, call-graph expansion, weight() constants, the add_reason max-merge, and the token budget are unchanged — tiering supersedes the cross-scale comparison, so this is a ranking-only change. (A future overhaul could normalize the semantic/graph scales and add a lexical path boost; noted, not done here.)

Tests

The existing ranking tests only used single-tier Explicit reasons, so they never exercised the semantic-vs-graph inversion. Added:

  • test_semantic_match_ranks_above_graph_only — a SemanticMatch{0.5} file must outrank a CalledBy{depth:1} (0.8) file.
  • test_semantic_tier_orders_by_score_then_path — intra-tier score order, path tie-break.
  • test_rank_files_is_deterministic — same set in any input order → identical ranked paths.

Verification

  • Full suite green: 381 tests, 0 failures; cargo fmt --check and cargo clippy --all-targets -D warnings clean.
  • End-to-end against a real index: ctx smart "add a new output format to ctx sql" run 5× now returns the identical set every time, with src/commands/sql.rs at rank 1. Spot-checks for "parse solidity contracts" and "generate embeddings with openai" surface the relevant implementation files (parser/mod.rs, embeddings/mod.rs) and are likewise deterministic.

`ctx smart <task>` selected the wrong files and was non-deterministic. For
"add a new output format to ctx sql" it returned generic hubs (main.rs,
smart_cmd.rs, analysis.rs) and unrelated test files while omitting
src/commands/sql.rs — even though that file's symbols are the top
`ctx semantic` hits — and the selected set changed on every run.

Two defects in rank_files (src/smart.rs):

1. Scale inversion. A direct SemanticMatch is scored with the raw
   similarity, which the 1/(1+L2) conversion compresses to ~0.4-0.6, while
   every depth-1 call-graph neighbor gets a flat 0.8 (CalledBy) / 0.7
   (Calls). So a file pulled in purely by call-graph expansion outranked the
   on-topic seed that produced it, then consumed the token budget first
   (select_by_token_budget is greedy first-fit) and evicted the seed.

2. Non-determinism. rank_files sorted by relevance_score alone with no
   tie-break, over a Vec collected from a HashMap. The many files tied at
   0.8/0.7/0.5 came out in hash order, so which tied files survived the
   budget varied per run.

Rework rank_files into a tiered, total ordering:
  (1) files with a direct semantic match rank above graph-only files
      (via new FileSelection::best_semantic_score), so call-graph expansion
      supplements the seeds instead of displacing them;
  (2) within a tier, higher score first (semantic score for the semantic
      tier, relevance_score for the graph-only tier — never compared across
      scales);
  (3) ties break by path ascending, making selection deterministic.

Seeding, call-graph expansion, weight() constants, the add_reason max-merge,
and the token budget are all unchanged — tiering supersedes the cross-scale
comparison, so this is a ranking-only change.

Add unit tests for the tier ordering, intra-tier score/path ordering, and
determinism (same set in any input order yields the same ranked paths). The
existing tests only used single-tier Explicit reasons, so they never
exercised the semantic-vs-graph inversion.

Verified end-to-end: the query above now deterministically selects
src/commands/sql.rs at rank 1 across repeated runs.
@saldestechnology
saldestechnology merged commit 91707b7 into main Jul 10, 2026
5 checks passed
@saldestechnology
saldestechnology deleted the worktree-fix-smart-relevance branch July 11, 2026 13:04
saldestechnology added a commit that referenced this pull request Jul 11, 2026
Two residual `ctx smart` misses remained after #18 (which made selection
deterministic and semantic-first), each with a distinct cause:

- A candidate whose path literally contains a task word could still be
  outranked by the semantic matches and squeezed out by the token budget
  (e.g. `embeddings/openai.rs` for "…openai" — present only via a call-graph
  edge). A lexical signal fixes this.
- The single most-relevant file could exceed the whole budget and be dropped
  by greedy first-fit, which then backfilled with smaller, less-relevant
  files (e.g. `parser/solidity.rs`, 9074 tokens > the 8000 budget, was
  already rank 1 yet vanished). A budget-policy fix is needed.

Part A — lexical path boost:
- New shared `utils::lexical_tokens`: lowercases, splits on non-alphanumeric
  and camelCase boundaries, drops length-≤-1 tokens and a small stopword set.
- `FileSelection` gains `lexical_score` = count of distinct task tokens that
  appear in the file's PATH (path only, not symbol names: matching identifiers
  would let ubiquitous names like `ctx` — a test helper across `tests/*_cli.rs`
  — score a hit on nearly every file and drown out the on-topic one).
- `rank_files` treats a lexical hit as top-tier alongside semantic matches and
  orders lexical hits first within the tier — surfacing the on-topic file
  without any tunable float weight. Determinism (path tie-break) is preserved.

Part B — budget guarantee:
- `select_with_guaranteed_top` always includes the rank-1 file even when it
  alone exceeds the budget, then first-fits the remainder (reusing
  `select_by_token_budget`). The handler prints a one-line stderr note when the
  top file is oversized.

Adds unit tests for the tokenizer, lexical promotion/ordering, path-only
scoring (asserting `ctx` in a symbol name does not match), and the oversized-
top budget policy. #18's determinism/tiering tests still pass with
`lexical_score = 0`.

Verified end-to-end: "…openai" now selects `embeddings/openai.rs` at rank 1;
"parse solidity contracts" includes `parser/solidity.rs` with the oversize
note; "add a new output format to ctx sql" keeps `src/commands/sql.rs` at
rank 1 and stays deterministic across runs.
saldestechnology added a commit that referenced this pull request Jul 11, 2026
…rt (#25)

* feat: lexical path boost + oversized-top budget guarantee for ctx smart

Two residual `ctx smart` misses remained after #18 (which made selection
deterministic and semantic-first), each with a distinct cause:

- A candidate whose path literally contains a task word could still be
  outranked by the semantic matches and squeezed out by the token budget
  (e.g. `embeddings/openai.rs` for "…openai" — present only via a call-graph
  edge). A lexical signal fixes this.
- The single most-relevant file could exceed the whole budget and be dropped
  by greedy first-fit, which then backfilled with smaller, less-relevant
  files (e.g. `parser/solidity.rs`, 9074 tokens > the 8000 budget, was
  already rank 1 yet vanished). A budget-policy fix is needed.

Part A — lexical path boost:
- New shared `utils::lexical_tokens`: lowercases, splits on non-alphanumeric
  and camelCase boundaries, drops length-≤-1 tokens and a small stopword set.
- `FileSelection` gains `lexical_score` = count of distinct task tokens that
  appear in the file's PATH (path only, not symbol names: matching identifiers
  would let ubiquitous names like `ctx` — a test helper across `tests/*_cli.rs`
  — score a hit on nearly every file and drown out the on-topic one).
- `rank_files` treats a lexical hit as top-tier alongside semantic matches and
  orders lexical hits first within the tier — surfacing the on-topic file
  without any tunable float weight. Determinism (path tie-break) is preserved.

Part B — budget guarantee:
- `select_with_guaranteed_top` always includes the rank-1 file even when it
  alone exceeds the budget, then first-fits the remainder (reusing
  `select_by_token_budget`). The handler prints a one-line stderr note when the
  top file is oversized.

Adds unit tests for the tokenizer, lexical promotion/ordering, path-only
scoring (asserting `ctx` in a symbol name does not match), and the oversized-
top budget policy. #18's determinism/tiering tests still pass with
`lexical_score = 0`.

Verified end-to-end: "…openai" now selects `embeddings/openai.rs` at rank 1;
"parse solidity contracts" includes `parser/solidity.rs` with the oversize
note; "add a new output format to ctx sql" keeps `src/commands/sql.rs` at
rank 1 and stays deterministic across runs.

* test: deterministic offline regression guard for ctx smart ranking

Adds tests/smart_relevance.rs — a library-level integration test that guards
the smart file-selection ranking against regressions without needing the
fastembed model (which downloads ~90MB and is not byte-stable across arches,
so it cannot run in CI).

It hand-builds a tiny on-disk index (files + symbols + edges) and seeds exact
embedding vectors via the library API (Database::store_embedding), then calls
smart_context_with_embedding directly with a crafted query vector — exercising
the full seed -> call-graph expand -> lexical -> rank -> budget pipeline
deterministically and offline.

Two scenarios:
- path_match_beats_symbol_name_noise: a task token in a file's PATH must
  promote the on-topic file over a higher-semantic-scored file whose only match
  is a symbol NAME. This reproduces the `ctx` test-helper regression that was
  originally caught by hand. Verified as a real guard via mutation: reverting
  compute_lexical_score to also match symbol names makes this test fail.
- graph_only_path_match_is_promoted: a candidate reachable only through
  call-graph expansion whose path matches the task is surfaced to the top
  rather than being outranked by the semantic matches and dropped.

* fix: MCP analysis config variable shadowed SmartConfig under --all-features

The project-config load added in #28 was named `config`, shadowing the
`SmartConfig config` above it, so `smart_context_with_embedding(..., config)`
passed the wrong type. Only compiled under the `mcp` feature, so the default
build (and macOS CI) missed it; `--all-features` failed to compile. Rename the
project config to `project_config`. (Latent on main from #28; this fix lands
there via this PR.)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant