feat(search): hybrid BM25 + vector ranking via RRF (promote lexical to first-class)

[HumanBean17]

Summary

Fold a LadybugDB FTS BM25 score into the vector/hybrid search read path as an additional RRF list, so exact-identifier lexical matches anchor dense rankings on the primary path (Apple Silicon / Linux / Windows) — not just the macOS-Intel fallback where lexical currently lives. Prerequisite: fork A (separate PR) builds the search_text column + FTS index on the Symbol graph.

Background

• PR feat(search): lexical fallback for graph-only / macOS Intel (PR-LEX) #403 shipped a lexical fallback that runs only where the vector stack can't install (macOS Intel: torch>=2.3 / lancedb>=0.26 dropped x86_64 wheels). It ranks by hand-rolled token overlap (name > fqn > signature), capped at a 5000-symbol Python-side scan (search_lexical.py:_CANDIDATE_LIMIT_CAP).
• The PR notes said "upgradable to SQLite FTS5 later." That pointed at the wrong store. The graph DB is LadybugDB — the embedded Cypher/GNN store that is kuzu's ancestor (not a kuzu wrapper), engine v0.17.1. LadybugDB ships a native FTS extension that scores with Okapi BM25, fetched from its own registry at extension.ladybugdb.com. Same code_graph.lbug the graph already lives in — no new dependency, no sidecar store. (Verified on-disk: INSTALL FTS → LOAD EXTENSION FTS → CREATE_FTS_INDEX → QUERY_FTS_INDEX returns BM25 scores.)
• Fork A uses that to make the fallback production-grade (DB-side ranking kills the 5000-row cap). This issue (B) is the follow-on: stop wasting that BM25 signal on the fallback path alone.

Proposal

The vector path already does 2-list RRF (vector + graph-expand) in search_lancedb.py (_HYBRID_SCORE_MAX = 2/(k+1), k=60). Add the Symbol BM25 score as a third RRF list:

final_rank = RRF([vector_hits, graph_expand_hits, bm25_hits])

• BM25 list sourced from the same LadybugDB FTS index built in fork A (CALL QUERY_FTS_INDEX('Symbol','sym_fts', $q, K:=1.2, B:=0.75, top:=$k) over Symbol.search_text).
• The pure-lexical path (Intel) becomes "hybrid with an empty vector list" — one read path, not two.
• No tool-contract change (SearchHit / SearchOutput / NodeFilter unchanged); _score_components gains a bm25 entry; explain_score_components already has a lexical= flag to extend.

Why lexical earns its keep on the primary path

Code search is unusually kind to lexical methods: identifiers are exact-match signals (DistributionChunkService, processClientMessage). Dense embeddings approximate that signal; BM25 nails it. Anchoring dense rankings with exact-identifier BM25 is the standard hybrid win for code retrieval and consistently beats either method alone. Long-term option this unlocks: an optional --no-vectors mode (drop the torch/lancedb/sentence-transformers dependency monster) for environments where pure-lexical quality is "good enough." Not in scope here — but B is the prerequisite that makes it reachable.

Scope

• In: third RRF list from the Symbol FTS index; unified read path; quality eval (recall/precision @k vs current 2-list hybrid) on ~/jrag-bench/shopizer.
• Out: sql / yaml Lance tables (no LadybugDB FTS there — different store); a vector-free distribution; replacing vectors.

Depends on

Fork A — BM25 lexical fallback via LadybugDB FTS — builds the search_text column + FTS index this issue consumes. Tracked as a separate PR.

Open questions

1. RRF k for a 3-list fusion (2-list uses k=60) — tune on the shopizer eval, or keep 60?
2. Should the BM25 list also feed absence_diagnosis (empty-result disambiguation), or stay rank-only?
3. Latency budget for the extra QUERY_FTS_INDEX hop on the primary path (it's a DB-side indexed query, but it is a new call per search)?
4. Offline/airgapped: INSTALL FTS fetches from extension.ladybugdb.com on first use (cached after). Need a bundled/local-path install path for airgapped operators — shared with fork A.

🤖 Generated with Claude Code