perf: prefilter BM25 channel candidates with Postgres FTS

[nuemaan]

Summary

• path_channel and content_channel loaded every scoped chunk with non-empty search text into Python before ranking, so cost grew with namespace size on every retrieve step. term_channel already pushed its query filter into SQL; these two did not, even though the GIN-indexed content_search_tsv and path_search_tsv columns were already selected by the scoped-corpus CTE.
• Both channels now prefilter on the tsvector column and pass a bounded candidate pool to the existing Python BM25 ranker, which stays the final per-channel ranker. ts_rank_cd only orders which candidates survive the limit.
• New setting RETRIEVAL_POSTGRES_FTS_CANDIDATE_LIMIT, default 2000.
• No API, worker, deployment, or migration impact. Response shape, RRF fusion, citations, result assembly, and workflow/agentic routing are untouched. The tsvector columns and GIN indexes already exist, so no migration is needed.
• Resolves Optimize BM25 retrieval candidate loading with Postgres FTS prefilter #195

On tsquery construction: the query is built inside SQL from a text[] of ranker tokens. Postgres lexes each token with the same simple configuration that generates the tsvector columns, then ORs the resulting lexemes. Two reasons for doing it server-side rather than assembling tsquery text in Python:

1. It stays aligned with the stored lexicon. The simple parser splits foo_bar into two lexemes but keeps a.b.c whole, and lexes gpt-4 as gpt and -4. Reproducing those rules in Python would drift from what is actually indexed.
2. Operators in user input stay data. A query containing &, |, or ! cannot change the shape of the filter.

OR semantics match how the Python ranker admits a row on any token intersection, so the prefilter cannot narrow recall relative to current behaviour. When no token yields a lexeme, or the prefilter matches nothing, the channel falls back to the previous unfiltered scan.

Verification

Commands run:

• uv run python -m pytest apps/api/tests/unit/test_bm25_channel_tsquery.py — 8 passed
• uv run python -m pytest apps/api/tests/contract/test_bm25_fts_prefilter_contract.py — 7 passed, against a real Postgres 16
• uvx ruff check on all changed files — clean

The contract tests run against real Postgres rather than a fake, because a fake would not catch a mismatch between the query configuration and the one the generated columns use. They cover the English and CJK paths, OR semantics across tokens, exclusions still applying under the prefilter, the no-match fallback, and that tsquery operators in a query do not change the filter shape.

Measured on a seeded corpus of 500 scoped chunks where 2 match the query:

before (full scoped scan):   500 rows loaded into Python
after  (FTS prefilter):        2 rows loaded into Python

Not tested: production-scale corpora, and the interaction with agentic multi-step retrieval under load. Both would need a dataset I do not have locally.

Deployment Notes

• New optional env var RETRIEVAL_POSTGRES_FTS_CANDIDATE_LIMIT (int, default 2000). Existing deployments need no change.
• No database migrations. content_search_tsv, path_search_tsv, and their GIN indexes were added in d4e5f6a7b8c9_add_checkerboard_search_fields.
• Backwards compatible. The fallback path preserves the previous behaviour exactly, so rolling back is a code-only revert.

Checklist

• Tests were added or updated when behavior changed
• Public docs, examples, or OpenAPI contracts were updated when needed
• Database migrations are idempotent and safe to deploy
• Logs, errors, and validation paths avoid leaking secrets or user data
• The pull request description explains any breaking or user-visible change

[EricNGOntos]

Thanks for this — classic BM25 candidate loading is a real issue we want to keep fixing (classic mode is a long-lived path even though map-nav is now the default).

We're landing a combined fix in #252 that keeps your strongest pieces:

• server-side OR tsquery built from text[] (lexeme-aligned with simple, operators stay data)
• full-scan fallback when FTS matches nothing (no silent recall regression)
• real Postgres contract coverage + token unit tests + RetrievalConfig

It also absorbs SQL-side exclude_sections before LIMIT from #244, and credits both of you with Co-authored-by.

Closing this PR as superseded by #252 so we don't merge two conflicting approaches for #195. Please keep contributing — follow-ups welcome (classic large-namespace soak tests, metrics, etc.).

[EricNGOntos]

Superseded by #252 (combined landing of #244 + #251 for classic BM25 FTS prefilter).