feat(search): subword tokens + Porter stemmer + stopword filter for FTS

[mschreib28]

Context\n\nEmbeddings were removed in 453c39d (issue colbymchenry#87). Since then, all search quality has to come from FTS. The original PR colbymchenry#74 documented several queries where FTS-only badly trailed semantic search — the dominant cause is that SQLite's default tokenizer treats getParser as one indivisible token, so a query for parser never matches it at the FTS layer no matter how clever the post-hoc rescorer is.\n\nThis PR closes most of that gap with three compounding, model-free changes.\n\n## What changed\n\n1. Subword tokens. New name_subwords column on nodes populated with the camel/snake split of the identifier (kept alongside the original) and indexed by FTS5 at weight 10×. A query for parser now finds getParser at the FTS layer, not just via post-hoc rescoring on the limited candidate set BM25 surfaces.\n\n2. Porter stemmer. tokenize="porter unicode61" on the FTS table collapses morphological variants — parser / parsing / parses all stem to pars so a natural-language query matches identifier subwords and docstring prose alike.\n\n3. Stopword stripping. searchNodesFTS filters stopwords from the query before the OR-join. Without this, words like how / does / the become OR'd FTS hits against any prose-bearing docstring and crowd out the actually-relevant identifier tokens. Reuses the existing STOP_WORDS set in src/search/query-utils.ts via a new shared filterStopwords helper (no second source of truth).\n\n## Empirical results\n\nBench: codegraph's own src/, 1242 nodes, 71 files. Same query set as PR colbymchenry#74's evidence. Lower rank = better.\n\n| Query | baseline (current main) | this PR |\n|---|---:|---:|\n| ExtractionOrchestrator | 1 | 1 |\n| how does file parsing work (target: getParser) | NOT FOUND in 20 | #2 |\n| database connection management (target: DatabaseConnection) | #18 | #1 |\n| resolves references between modules (target: resolveOne) | #19 | #2 |\n\nMean rank: ~14 → 1.5. Same workload, no model dependencies, no install-time changes, no Node version constraints.\n\n## What I tested and rejected\n\nConcept-mode docstring re-weighting (swap BM25 weights to favor docstring 20× when query contains stopwords): regressed how does file parsing work from #4 to #11 because amplifying docstring weight floods results with prose-keyword spam more than it lifts truly relevant prose. Not included.\n\n## Migration v4\n\nExisting v3 databases get migrated by:\n- Adding the name_subwords column to nodes (idempotent PRAGMA table_info guard so a re-run after partial DDL failure doesn't fail with "duplicate column")\n- Dropping the old FTS table + triggers (FTS5 tokenizer cannot be ALTERed)\n- Recreating FTS without triggers\n- Backfilling name_subwords for every existing node\n- Rebuilding the FTS index in one shot via INSERT INTO nodes_fts(nodes_fts) VALUES('rebuild')\n- Recreating the triggers afterward (so they don't fire mid-backfill — earlier prototype runs corrupted FTS5 when triggers fired against half-populated shadow tables)\n\nCURRENT_SCHEMA_VERSION 3 → 4. Two existing version-pinned tests updated.\n\n## Files changed\n\n| File | Change |\n|---|---|\n| src/utils.ts | Add splitIdentifierTokens, buildNameSubwords (with dedup) |\n| src/search/query-utils.ts | Add shared filterStopwords helper using existing STOP_WORDS |\n| src/db/schema.sql | Add name_subwords column, add it to nodes_fts, add tokenize="porter unicode61", update three triggers |\n| src/db/migrations.ts | Bump version to 4; add migration v4 with idempotent ALTER guard |\n| src/db/queries.ts | Populate name_subwords on insert/update; new BM25 weights (0,20,5,1,2,10); stopword filter in searchNodesFTS |\n| __tests__/foundation.test.ts, __tests__/pr19-improvements.test.ts | Update expected schema version |\n| __tests__/search-quality.test.ts (NEW) | 21 regression tests: helpers, end-to-end search, full v3-to-v4 migration with a real v3-shape DB, and migration idempotency on partial-DDL re-run |\n\n## Test plan\n\n- [x] npm test: 404/404 pass on macOS (one pre-existing fs.watch flake under parallel load, passes in isolation — unrelated)\n- [x] npx tsc --noEmit clean\n- [x] Bench script confirms the rank lifts in the table above\n- [x] Independent reviewer pass before pushing — addressed three findings:\n - merged duplicate stopword sets (now uses STOP_WORDS from query-utils.ts)\n - dedup tokens in buildNameSubwords (parse no longer stores "parse parse")\n - made migration idempotent on partial-DDL re-run (re-running after a crash that landed only ALTER TABLE no longer fails)\n\n## Notes for review\n\n- This PR is independent of colbymchenry#102 (which also bumps schema version to 4). If colbymchenry#102 lands first, I'll rebase to v5.\n- No new dependencies. No install-time changes. No Node version constraints.\n- The change is gradual — fresh installs hit the new schema directly via schema.sql, existing installs run migration v4 on first open.\n\n🤖 Generated with Claude Code\n

---

Copied from colbymchenry/codegraph#104