feat: lexical path boost + oversized-top budget guarantee for ctx smart

[saldestechnology]

Follow-up to #18. After #18 made ctx smart deterministic and semantic-first, two residual misses remained — and diagnosis (via ctx smart --dry-run, which lists the full candidate set) showed they have different root causes:

Case	Cause	Fix
embeddings/openai.rs missing for "generate embeddings with openai"	It's a candidate (tier-1 Calls) whose path contains "openai", but ranks below the semantic matches and is squeezed out by the budget	Lexical path boost
parser/solidity.rs missing for "parse solidity contracts"	Already rank-1 SemanticMatch, but 9074 tokens > the 8000 budget, so greedy first-fit drops it and backfills with smaller files	Budget guarantee

(The original #18 case — "add a new output format to ctx sql" → src/commands/sql.rs at rank 1 — must not regress.)

There is no relevance-eval harness in the repo to tune against (the merged perf/, snapshot, and run-record work are timing / code-metrics / agent-outcome tools, not file-selection evals), so this uses integer token-overlap counts — no float coefficient — and is validated against concrete queries + unit tests.

Part A — lexical path boost

• New shared utils::lexical_tokens: lowercases, splits on non-alphanumeric and camelCase boundaries, drops length-≤-1 tokens and a small stopword set.
• FileSelection.lexical_score = count of distinct task tokens present in the file's path. Path only, not symbol names — matching identifiers is low-precision: ctx (the tool's own name) appears as a test helper across tests/*_cli.rs and would score a hit on nearly every file, demoting the on-topic one. (This was caught during end-to-end verification: an earlier symbol-name variant regressed the sql case by promoting test files; path-only fixes it.)
• rank_files treats a lexical hit as top-tier alongside semantic matches and orders lexical hits first within the tier. fix: make ctx smart deterministic and semantic-first #18's determinism (path tie-break) and tiering are preserved.

Part B — budget guarantee

• select_with_guaranteed_top always includes the rank-1 file even when it alone exceeds the budget, then first-fits the remainder (reusing the shared select_by_token_budget). The handler emits a one-line stderr note when the top file is oversized.

Verification

• Full suite green: fmt, clippy --all-targets -D warnings, all tests pass (lib 305 incl. new tokenizer / lexical-promotion / path-only / budget tests; fix: make ctx smart deterministic and semantic-first #18 tests still pass with lexical_score = 0).
• End-to-end (release build vs. the live index):
  • "generate embeddings with openai" → src/embeddings/openai.rs at rank 1.
  • "parse solidity contracts" → src/parser/solidity.rs included, with the oversize stderr note.
  • "add a new output format to ctx sql" → src/commands/sql.rs at rank 1, identical set across 5 runs (no fix: make ctx smart deterministic and semantic-first #18 regression; the ctx-noise test files are correctly demoted).
• No perf/ impact: smart is not a perf scenario.

[saldestechnology]

Added tests/smart_relevance.rs — a deterministic, offline regression guard for the ranking, closing the gap that this bug (the ctx symbol-name noise) was originally caught by hand.

It can't drive the real CLI in CI (the fastembed model downloads ~90MB and isn't byte-stable across arches), so instead it hand-builds a tiny on-disk index and seeds exact embedding vectors via the library API, then calls smart_context_with_embedding with a crafted query — exercising the full seed → call-graph expand → lexical → rank → budget pipeline with no model download.

• path_match_beats_symbol_name_noise reproduces the exact regression: a task token in a file's path must outrank a higher-semantic file whose only match is a symbol named like a task token. Verified as a real guard by mutation — reverting compute_lexical_score to also match symbol names makes this test fail.
• graph_only_path_match_is_promoted guards the tier-promotion of a call-graph-only candidate whose path matches the task.