feat(search): ranked full-text paragraph search — tsvector, GET /search, MCP parity

[thewrz]

Why

Every downstream client of this headless API — an SPA, an LLM/agent workflow over MCP, a reporting integration — starts almost every task with "find the relevant paragraphs." The only deterministic retrieval was searchParagraphs, an ILIKE substring scan with no ranking, no stemming, and poor recall for natural-language queries. Retrieval quality caps the quality of everything built on top, so the deterministic layer should own it.

What

Ranked PostgreSQL full-text search replaces the ILIKE scan, exposed as a new REST route and brought to parity on MCP.

• Migration 042 — a STORED generated tsvector column on paragraphs (English config, from text) plus a GIN index. Generated + STORED keeps the vector consistent with the text with no trigger and no app write path; existing rows gain a populated vector with no backfill.
• searchParagraphs rewrite — websearch_to_tsquery + ts_rank_cd + ts_headline snippets, scoped by libraryId / projectId / division / part (CSI PART 1/2/3) / nodeType. A blank query returns []; a degenerate (no-lexeme) query falls back to an escaped ILIKE substring scan. Each hit carries stable anchors: paragraph UUID, spec id, section, title, plus snippet and rank.
• GET /search — the REST twin of the previously MCP-native search_library, documented in openapi.yaml (SearchHit schema) in the same PR.
• MCP search_library — same query path and the same scope filters, so REST and MCP return identical rows for identical inputs. snippet/rank are additive, so the response stays backward-compatible. contract-map now pairs get /search → search_library and both contract gates stay green.
• ADR-062 — records the retrieval stance: full-text search is in-core and deterministic; semantic similarity (embeddings/pgvector) is explicitly deferred to its own future decision.

Design decisions

Ambiguous calls made autonomously (issue was the spec; brainstorming skipped by request):

• ts_rank_cd over plain ts_rank. The acceptance criterion "exact-phrase beats scattered terms" is proximity-sensitive; cover-density ranking is the proximity-aware member of the ts_rank* family, so a tight cluster of the query terms outranks the same terms spread across a long paragraph. The issue named ts_rank generically.
• part filter = enclosing CSI PART ordinal (1/2/3). There is no stored part pointer, so each hit is climbed to its root paragraph via a recursive CTE and the spec's root parts are ranked by document order. This matches what a spec editor means by "search within Products."
• Search does not filter meta.vanish. Retrieval surfaces all stored content; suppression of hidden content is a render-layer concern (the markdown renderer), not a retrieval one. Filtering here would silently drop recall.
• Snippet delimiters <mark>…</mark>. Neutral highlight markup a UI can style and an agent can strip. Returned in a JSON field of a headless API — clients escape on render as usual.
• search_library stays in INV5_READ_PENDING. A non-vacuous response-shape assertion needs seeded paragraph content (a parsed spec) beyond pnpm seed, mirroring list_library_specs.
• toSearchOptions omit-undefined constructor. Lets the REST/MCP layers build the options object from validated-but-optional fields under exactOptionalPropertyTypes.
• Extracted library-tools.ts. Adding the parity filters pushed tools.ts over the enforced 400-line cap, so the library/search tool registrations moved to their own module (the established *-tools.ts pattern).

Testing

• Unit tests pass (pnpm test → 1633 passed)
• Touched integration tests pass (search db + api, both contract gates, MCP server) → 79 passed
• Migration up/down/up verified clean
• Lint + typecheck + format green (pnpm lint)
• Ranking regression: tight cluster outranks scattered terms; part/division/nodeType scope filters; blank + degenerate-fallback queries
• REST↔MCP parity via both contract gates (src/api/contract.integration.test.ts, src/mcp/contract.integration.test.ts)
• CI green

🤖 Co-authored by Claude Fable 5. Closes #445.

Summary by CodeRabbit

• New Features

  • Added ranked full-text paragraph search through the REST GET /search endpoint.
  • Added filtering by library, project, division, part, node type, and result limit.
  • Search results include navigation metadata, relevance ranking, and highlighted snippets.
  • Added matching MCP search_library functionality with equivalent search filters.

• Bug Fixes

  • Added safe handling for blank, punctuation-only, and stop-word-only searches.
  • Search snippets now safely escape HTML content.

• Documentation

  • Documented the project’s full-text retrieval approach and future semantic-search considerations.

[coderabbitai]

Important

Review skipped

Auto incremental reviews are disabled on this repository.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: f67a7a91-ddbd-4580-b866-101cd301b798

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

📝 Walkthrough

Walkthrough

Adds deterministic ranked PostgreSQL full-text paragraph search with generated vectors, REST GET /search, expanded MCP search_library filters, shared result shaping, contract mappings, and integration coverage for ranking, scoping, validation, fallback queries, and HTML-safe snippets.

Changes

Ranked paragraph search

Layer / File(s)	Summary
Retrieval stance and implementation plan docs/adr/..., docs/superpowers/plans/...	Documents PostgreSQL full-text retrieval, English-only behavior, fallback handling, REST/MCP parity, and implementation decisions.
FTS storage and query execution src/db/migrations/..., src/db/queries/..., src/db/index.ts	Adds the generated tsvector and GIN index, ranked SQL construction with scope filters and snippets, expanded search options, and regression coverage.
REST search endpoint and contract src/api/..., openapi.yaml	Adds validated GET /search, the SearchHit response schema, router wiring, and endpoint contract/integration tests.
MCP search parity and tool registration src/mcp/...	Extends search_library filters, routes them through shared search options, extracts library-tool registration, and maps the REST route to MCP contract coverage.

Estimated code review effort: 4 (Complex) | ~45 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Client
  participant RESTSearch
  participant MCPSearch
  participant searchParagraphs
  participant PostgreSQL
  Client->>RESTSearch: GET /search or search_library request
  RESTSearch->>searchParagraphs: validated query and filters
  MCPSearch->>searchParagraphs: converted query and filters
  searchParagraphs->>PostgreSQL: ranked FTS or fallback query
  PostgreSQL-->>searchParagraphs: hits with snippets and ranks
  searchParagraphs-->>RESTSearch: SearchHit results
  searchParagraphs-->>MCPSearch: SearchHit results

Loading

Possibly related PRs

• wrzonance/SpecR#24: Implements the related MCP search_library tool and handler flow.
• wrzonance/SpecR#330: Extends the same MCP search handler with search-result anchor metadata.
• wrzonance/SpecR#334: Introduces the REST/MCP contract and gate framework updated here.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 53.33% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title is concise and accurately summarizes the search, tsvector, REST, and MCP changes.
Linked Issues check	✅ Passed	The PR implements the migration, ranked search, REST/MCP parity, ADR, and tests required by #445.
Out of Scope Changes check	✅ Passed	Changes stay within the search feature, its docs, and tests; no unrelated scope appears introduced.

---

_{Comment @coderabbitai help to get the list of available commands.}

[thewrz]

@coderabbitai full review

[coderabbitai]

✅ Action performed

Full review finished.

---

You're currently rate limited under our Fair Usage Limits Policy. Your recent PR review activity is in the 95th percentile or higher among CodeRabbit users, so adaptive limits apply. Your next review will be available in 41 minutes.

[thewrz]

Codex adversarial review (review of record — CodeRabbit rate-limited)

CodeRabbit hit its Fair-Usage rate limit on this PR (next window ~50 min), so Codex (GPT-5.5, xhigh) ran as the review gate against origin/main. Two findings, both fixed in b52000b:

• [P2] Stored XSS in snippet — src/db/queries/search-query.ts. The snippet field carries <mark> highlight tags and is documented/rendered as HTML, but ts_headline and the ILIKE fallback (left(p.text, 200)) returned uploaded paragraph text verbatim, so text like <img onerror=…> would reach a consumer as live markup. Fixed: the source text is HTML-escaped (&,<,>) in SQL before the <mark> tags are inserted, so <mark> is the only live markup. Both snippet paths (ts_headline and ILIKE fallback) now have regression tests in search.integration.test.ts.

• [P3] nodeType contract drift — openapi.yaml. The handler validates nodeType against the 12-value NodeTypeSchema (400 on miss) while the OpenAPI parameter documented type: string (any string), so a spec-generated client could send values the route rejects. Fixed: published the matching enum on the parameter schema.

Verification: pnpm lint clean; full unit suite (1633) green; search/api-search integration + both contract gates (src/api/contract.integration.test.ts, src/mcp/contract.integration.test.ts) green.

[thewrz]

@coderabbitai review

[coderabbitai]

✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

[coderabbitai]

Actionable comments posted: 3

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@openapi.yaml`:
- Around line 233-236: Add maxItems: 100 to the array schema for data under the
response definition, alongside its type and items fields, so the documented
SearchHit result array matches the endpoint’s limit constraint.

In `@src/db/queries/search.ts`:
- Around line 44-47: Preserve backward compatibility for searchParagraphs by
supporting the previous positional arguments (query, division, limit) alongside
the new ParagraphSearchOptions object. Add overloads and an adapter in
searchParagraphs that normalize both call styles into the current options
format, ensuring existing TypeScript and JavaScript callers retain their
division and limit behavior.

In `@src/mcp/contract-map.ts`:
- Around line 212-214: Add a seeded parsed-paragraph fixture to the INV-5
contract test, then invoke both `search_library` and `GET /search` with the same
query and assert their returned rows are identical. Update the
`INV5_READ_PENDING` handling for `search_library` so this parity assertion is
executed rather than merely listed.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: e84430b8-7302-4499-baa5-c714d6c53303

📥 Commits

Reviewing files that changed from the base of the PR and between ebfc988 and b52000b.

📒 Files selected for processing (17)

• docs/adr/062-full-text-retrieval-stance.md
• docs/superpowers/plans/2026-07-10-fts-paragraph-search.md
• openapi.yaml
• src/api/contract.integration.test.ts
• src/api/router.ts
• src/api/search.integration.test.ts
• src/api/search.ts
• src/db/index.ts
• src/db/migrations/042_paragraphs_search_vector.ts
• src/db/queries/search-query.ts
• src/db/queries/search.integration.test.ts
• src/db/queries/search.ts
• src/mcp/anchors.test.ts
• src/mcp/contract-map.ts
• src/mcp/handlers.ts
• src/mcp/library-tools.ts
• src/mcp/tools.ts