feat(6.24): Clio FTS title boosting (Cerefox parity)

[fstamatelopoulos]

Summary

Brings Clio's FTS ranking up to Cerefox parity for title-boosted search:
query terms that match a document title or a chunk heading now outrank
body-only matches at the same ratio Cerefox uses (effective A:B = 2.5×).

Migrates the existing 2-column clio_chunks_fts (chunk_title +
content) to a 3-column shape (doc_title + chunk_title + content),
populated automatically via SQLite triggers and weighted at query time
through SQLite FTS5's per-column bm25() knob.

Cerefox → Clio mapping

Concern	Cerefox (Postgres)	Clio (SQLite FTS5)
Mechanism	setweight(...,'A') || setweight(...,'B') baked into tsvector at insert time	3-column FTS5 + bm25(table, w0, w1, w2) weights at query time
Title:content ratio	A:B = 1.0:0.4 ≈ 2.5× via ts_rank_cd defaults	2.5× via bm25(2.5, 2.5, 1.0)
Title rename refresh	cerefox_update_chunk_fts(doc_id, new_title) RPC, called from app code	clio_documents_fts_title_au trigger, fires automatically on UPDATE OF title
Backfill	manual scripts/reindex_all.py after migration	automatic INSERT...SELECT inside the migration

The underlying ranker is necessarily different (Postgres ts_rank_cd
cover-density vs SQLite bm25 Okapi BM25) — that's a stack consequence,
not a tuning choice. Within each stack we apply the same conceptual
title-boost ratio. For typical agent retrieval (top-3 hits, small-to-mid
corpora) the ranker difference is barely visible; mid-list shuffling
that BM25's lack of proximity sensitivity introduces is hidden by hybrid
mode (semantic + FTS) once an embedder is installed.

Files

Added:

• packages/core/src/clio/migrations/0002_fts_title_boost.sql — drops
  the old 2-column FTS table + its triggers, recreates as 3-column,
  backfills via JOIN, reinstalls 5 triggers covering every path:
  • clio_chunks_fts_ai/_ad/_au — chunk lifecycle, JOIN to docs for
    doc_title
  • clio_documents_fts_bd (BEFORE DELETE) — pre-clears FTS for
    current chunks before the cascade-delete fires; required because
    the chunk_ad trigger can't reconstruct doc_title via JOIN once
    the doc row is gone
  • clio_documents_fts_title_au (AFTER UPDATE OF title) — refreshes
    FTS doc_title for all current chunks of the renamed doc;
    mirrors Cerefox's cerefox_update_chunk_fts RPC

Changed:

• packages/core/src/clio/db.ts — registers migration 0002.
• packages/core/src/clio/backend/local-clio.ts — both bm25(...) call
  sites (chunk-level path in searchFts + chunk-level path in the
  hybrid SQL) take per-column weights. Constants extracted as
  FTS_BM25_WEIGHT_DOC_TITLE / CHUNK_TITLE / CONTENT for one-spot
  tuning if the ratio ever needs adjusting.
• packages/core/src/clio/backend/local-clio.test.ts — 6 new tests
  (90 total, was 84): title-only outranks body-only, chunk-heading
  outranks body-only, rename refreshes FTS, rename doesn't blow away
  unrelated docs, soft-delete still removes from search, end-to-end
  backfill verification.

Drive-by fix:

• packages/server/src/routes/update.test.ts — stale assertion from
  6.20 (the URL-removal-for-security pass) updated to assert the
  field's absence; was quietly failing the server suite and got
  caught running tests for this work.

Plan housekeeping (in this PR):

• Row 6.24 marked ✅ with shipped-state notes.
• Iter-6 active-set + headline summary lines updated.
• New backlog row F.19 captures the two known FTS gaps deferred from
  this work: stop-word handling + Google-style query operators (both
  noisier/dumber than Cerefox-on-Postgres; both pure-TS query-time
  fixes). Trigger criteria documented; not in scope here.

Migration behaviour

• Runs lazily on first getClioBackend() call after server boot — i.e.
  the next time anything touches Clio (Memory page stats, search,
  ingest, …). Sub-second on small corpora; sub-minute on 10k chunks.
• Wrapped in BEGIN IMMEDIATE; ...; COMMIT per the migration runner —
  partial failures roll back cleanly, leaving the old 2-column table
  in place.
• Existing chunks pick up doc_title automatically via the
  INSERT...SELECT backfill — no separate cfcf clio reindex --refts
  command needed (the original 6.24 scope mentioned one; the in-place
  backfill made it unnecessary).

Test plan

• bun run typecheck clean
• bun run test (585 / 100 / 66 / 9 — all green; +6 new in
  local-clio.test.ts; server-suite glob is silent under bun-test as
  noted in prior PRs, individual files verified)
• Manual: ingest two docs, one with the search term in the title
  only, one in the body only — title-doc ranks first
• Manual: rename a doc to add a previously-unfindable term to its
  title, re-search — doc now appears
• Reviewer to verify the migration runs cleanly on their existing
  ~/.cfcf/clio.db (sub-second expected) and post-migration searches
  show titled docs ranked above body-only hits