feat(search): add hybrid search with RRF fusion

[aliasunder]

Summary

• Add hybrid search (FTS5 + sqlite-vec vector + RRF fusion) to vault_search tool
• Transparent upgrade: uses hybrid ranking when embeddings are available, falls back to FTS-only when they are not
• Three new functions: computeRrfScores (pure RRF), vectorSearch (KNN), hybridSearch (orchestrator)
• Updated vault_search tool description to reflect hybrid search capabilities

Spike result

8/9 previously-failed queries fixed, 0 regressions, ~8ms latency (RRF-only mode).

Test plan

• 9 computeRrfScores unit tests (table-driven: single/both systems, bonuses, custom k, sort order)
• 12 hybridSearch integration tests (fallback, hybrid ranking, filters, limits, dedup, callouts)
• All 1177 tests pass
• Build and lint clean
• Deploy to Lightsail and run 9-query eval against live vault

🤖 Generated with Claude Code

https://claude.ai/code/session_018ZbaHwP28ASYTpQnZk6MqM

Summary by CodeRabbit

• New Features
  • Search now uses a hybrid ranking approach that blends keyword and semantic matching for more relevant results.
  • Results can now fall back to keyword-only search when embeddings aren’t available.
  • Search snippets and result metadata are more complete for semantic matches, with improved relevance ordering.

[aliasunder]

@CodeRabbit review

[coderabbitai]

✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

[coderabbitai]

📝 Walkthrough

Walkthrough

Adds a hybrid search path to search-index.ts combining FTS5 keyword results with sqlite-vec KNN vector results via Reciprocal Rank Fusion (RRF), with automatic fallback to FTS-only when embeddings are unavailable. The MCP vault_search tool handler is updated to call hybridSearch instead of fullTextSearch, and ~770 lines of new tests cover the RRF logic and hybrid search behaviors.

Changes

Hybrid FTS+Vector Search (RRF)

Layer / File(s)	Summary
RRF scoring, VectorHit type, and SQL statements src/vault-mcp/search/search-index.ts	Exports computeRrfScores for fusing two ranked lists (reciprocal rank + top-rank bonus). Adds prepared SQL for KNN over note_vectors/note_chunks and note metadata lookup. Fixes embedding Buffer construction to use explicit byteOffset/byteLength.
vectorSearch and hybridSearch implementation src/vault-mcp/search/search-index.ts	Adds vectorSearch (embed query → KNN → deduplicate per note → return [] on failure). Adds hybridSearch (inflated FTS + vectorSearch → RRF merge → filter vector-only rows via notePassesFilters → cap by limit). Adds buildSnippetFromChunkText helper. Wires hybridSearch into the returned API object.
MCP tool description and handler update src/vault-mcp/mcp-core/tools/search-tools.ts	Updates VAULT_SEARCH tool description to document hybrid RRF semantics, fallback behavior, revised example and Returns/When-to-use docs. Switches handler call from fullTextSearch to hybridSearch.
Tests for computeRrfScores and hybridSearch src/vault-mcp/search/__tests__/search-index.test.ts	Adds computeRrfScores test block (scoring, rank bonuses, custom k, empty/disjoint inputs) and a comprehensive hybridSearch suite covering fallback modes, fusion ranking, vector-only metadata/snippets, all filter types, deduplication, limit enforcement, and snippet truncation.

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~60 minutes

Possibly related PRs

• aliasunder/vault-cortex#215: Adds sqlite-vec extension loading and initial vector search support in search-index.ts, which this PR builds the hybrid RRF path on top of.
• aliasunder/vault-cortex#217: Adds the embedding pipeline, vector tables, and embedNote wiring in search-index.ts that vectorSearch in this PR consumes.
• aliasunder/vault-cortex#218: Modifies the background embedding pass and rebuild API in search-index.ts, which controls when vector results become available to the hybrid search introduced here.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly matches the main change: adding hybrid search with RRF fusion.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch phase2-hybrid-search

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 3

🧹 Nitpick comments (1)

src/vault-mcp/search/search-index.ts (1)

871-878: 🩺 Stability & Availability | 🔵 Trivial | 🏗️ Heavy lift

Replace changed as assertions with runtime row guards.

These new DB/JSON paths assert row shapes instead of narrowing them, which violates the TypeScript guideline and can crash hybrid search if a row or persisted JSON value is malformed. Add small validators for vector rows, note rows, and parsed string arrays before using them.

As per coding guidelines, “Do not use any, as, or non-null assertions (!); use runtime guards or schema validation to narrow types instead.”

Also applies to: 1303-1324, 1896-1916

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/vault-mcp/search/search-index.ts` around lines 871 - 878, The hybrid
search and persisted JSON handling currently rely on `as` assertions instead of
validating the data shape at runtime, which can break on malformed rows or JSON.
Replace the assertions in the search flow (including the `knnSearchStmt.all`
result and the related DB/JSON paths around the other affected blocks) with
small runtime guards or schema checks that narrow rows to the expected vector
row, note row, and string-array shapes before use. Keep the existing search
logic in place, but make the row parsing explicit and safe by validating the
fields first and only proceeding with typed values after the guard passes.

Source: Coding guidelines

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/vault-mcp/search/__tests__/search-index.test.ts`:
- Around line 3328-3366: The vector-only search test is too weak because it does
not prove the snippet comes from the intended note. Update the mocked embedText
behavior in search-index.test.ts so each embed call maps deterministically to
the intended note/query embedding, then assert the returned result includes the
expected note path and the exact snippet value from that note’s chunk text. Use
the existing hybridSearch, embedNote, and createSearchIndex flow to keep the
test focused on the intended vector-only behavior.
- Around line 3748-3761: The truncated snippet assertion in the hybrid search
test is too loose even though the output is deterministic for this fixture. In
the `hybridIndex.hybridSearch` test case that inspects `verboseResult`, replace
the token-count/contains checks with a whole-value assertion against the exact
snippet produced by `buildSnippetFromChunkText`, so the test verifies the
precise truncated string for `verbose.md` rather than any five-word output with
ellipsis.

In `@src/vault-mcp/search/search-index.ts`:
- Around line 1271-1284: The RRF merge in search-index.ts is combining
unfiltered vectorHits from vectorSearch with ftsResults before applying
folder/tag/property constraints, so matching semantic results can be missed and
non-matching vectors can skew scores. Update the vector path in the search flow
around vectorSearch, computeRrfScores, and the later vector-only filtering to
apply the same filters to vector candidates before ranking, or pass the filters
into vectorSearch so only eligible notes contribute to RRF.

---

Nitpick comments:
In `@src/vault-mcp/search/search-index.ts`:
- Around line 871-878: The hybrid search and persisted JSON handling currently
rely on `as` assertions instead of validating the data shape at runtime, which
can break on malformed rows or JSON. Replace the assertions in the search flow
(including the `knnSearchStmt.all` result and the related DB/JSON paths around
the other affected blocks) with small runtime guards or schema checks that
narrow rows to the expected vector row, note row, and string-array shapes before
use. Keep the existing search logic in place, but make the row parsing explicit
and safe by validating the fields first and only proceeding with typed values
after the guard passes.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: e50beed6-d73a-4c02-8e62-79c731690a67

📥 Commits

Reviewing files that changed from the base of the PR and between c2a2a59 and 8e696da.

📒 Files selected for processing (3)

• src/vault-mcp/mcp-core/tools/search-tools.ts
• src/vault-mcp/search/__tests__/search-index.test.ts
• src/vault-mcp/search/search-index.ts