feat(retrieval): hybrid search with Vectorize + D1 FTS5 and RRF fusion - #86
Merged
liplus-lin-lay merged 1 commit intoApr 19, 2026
Conversation
Replace dense-only retrieval with DIY hybrid search: Vectorize BGE-M3 embeddings on the dense side and D1 FTS5 BM25 on the sparse side, combined via Reciprocal Rank Fusion (k=60). - Add D1 binding (DB_FTS) and migrations/0001_fts5_init.sql providing search_docs content table + two FTS5 virtual tables (porter tokenizer for natural language, trigram for diffs). Triggers fan out automatically so DELETE on search_docs clears both virtual tables. - New src/fts.ts: upsert/delete helpers, BM25 query across both tokenizers, and the reciprocal_rank_fusion utility. - pipeline.ts: mirror every Vectorize upsert (issue/PR/release/doc/diff) into D1 FTS5 with the matching tokenizer_kind. FTS5 failures are logged but never invalidate a successful Vectorize write. - webhook/poller delete paths fan out to D1 FTS5 for issue/PR/release/doc; diffs remain append-only by design. - mcp.ts search_issues runs dense and sparse queries in parallel and fuses them via RRF. Adds a fusion parameter (rrf / dense_only / sparse_only) for debugging and surfaces per-result dense_score, sparse_score, dense_rank, sparse_rank alongside the fused score. - admin reset-hashes now also clears search_docs for the given repo. - docs (ja + en) updated with hybrid retrieval architecture, D1 FTS5 / BM25 / RRF sections, fusion toggle, and Free tier hard-stop guarantees. - installation docs (ja + en) now describe `wrangler d1 create` and `wrangler d1 migrations apply` steps for DB_FTS. dense-only retrieval is known to lose recall on short identifiers, SHA prefixes, and exact-term queries. Hybrid BM25 + dense is the 2026 production baseline and directly targets the Diff RAG judgment-history surface where commit messages, file paths, and SHAs dominate. Refs #85 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Deploying with
|
| Status | Name | Latest Commit | Updated (UTC) |
|---|---|---|---|
| ❌ Deployment failed View logs |
github-rag-mcp | 316899a | Apr 19 2026, 11:06 AM |
liplus-lin-lay
deleted the
85-enhancement-hybrid-retrieval-vectorize-+-d1-fts5-with-rrf-fusion
branch
April 19, 2026 11:08
liplus-lin-lay
added a commit
that referenced
this pull request
Apr 19, 2026
Refs #87 PR #86 で導入した D1 binding の database_id placeholder を 実際に作成済みの D1 database (github-rag-fts, APAC region) の ID に差替え。 これで Cloudflare Workers Builds が placeholder で落ちる状態を解消する。 The D1 migration 0001_fts5_init.sql has already been applied to the remote database via `wrangler d1 execute --file` (since the multi-statement BEGIN...END triggers trip up wrangler's migration splitter). Migration state is recorded in the d1_migrations tracking table.
This was referenced Apr 19, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Refs #85
概要
Dense-only retrieval を hybrid search (dense + sparse BM25) に置換します。Vectorize BGE-M3 (dense) と D1 FTS5 BM25 (sparse) を Reciprocal Rank Fusion (k=60) で合成し、特に Diff RAG における judgment-history retrieval の recall を引き上げることが狙いです。
主な変更
wrangler.toml):DB_FTS(github-rag-fts) +migrations_dir = "migrations"migrations/0001_fts5_init.sql):search_docscontent table + 2 つの FTS5 virtual table (porter + unicode61自然言語用 /trigramコード・SHA・identifier 用) + trigger で FTS5 側の自動同期search_issues: dense (Vectorize) と sparse (D1 FTS5 BM25) を並列 query → RRF で rank 合成 → label / assignee post-filter。fusionパラメータでrrf(default) /dense_only/sparse_onlyを切替可能。result にdense_score/sparse_score/dense_rank/sparse_rankを追加、top-level にfusion/dense_candidates/sparse_candidates/admin/reset-hashesでsearch_docsの該当 repo 行も削除fusionパラメータを追記wrangler d1 create github-rag-ftsとwrangler d1 migrations apply手順を追加後方互換性
search_issuesのscoreの意味が cosine similarity (dense only) から RRF fused score に変わります。dense_scoreで従来の cosine 値を別途参照可能dense_score/sparse_score/dense_rank/sparse_rank) は non-breakingデプロイ前に必要な手作業
wrangler d1 create github-rag-ftsで D1 database を作成database_idをwrangler.tomlの placeholder (REPLACE_WITH_D1_DATABASE_ID) に差し替えwrangler d1 migrations apply github-rag-fts --remoteで初回 migration を適用wrangler deploy後、必要なら/admin/reset-hashes?repo=…で既存 repo を reindex動作確認
npx tsc --noEmit: passnpx wrangler deploy --dry-run: pass (env.DB_FTS (github-rag-fts)が bindings に含まれることを確認)注意
database_idは placeholder のまま commit しています。CI/CD が D1 database_id 実在性を validate する場合、最初のデプロイ前に Cloudflare 側で database を作成して ID を埋める必要があります。Milestone
v0.6.0
🤖 Generated with Claude Code