fix(hts): incremental sqlite concept FTS (stop full rebuild on every boot) (#295)

[mauripunzueta]

Summary

Fixes #295. The sqlite HTS server rebuilt the entire concept FTS index (concepts_fts, concepts_word_fts, concepts_search_fts) over all concepts on every startup, synchronously before TcpListener::bind(). finalize_after_bootstrap() did an unconditional DELETE of all FTS + prebuild_concepts_fts over the full corpus. With a full terminology corpus this exceeds the readiness window in a debug build, so the server never binds and /health (which just returns 200 once bound) never answers — breaking the sqlite HTS benchmark leg. concepts_search_fts (added in #273) pushed it over the edge.

Root cause is architectural: FTS was the one index without the invalidate-on-import → incremental-startup-rebuild → prebuild-in-CLI-import lifecycle that concept closures already have. This PR gives FTS that same lifecycle, so the per-boot rebuild is removed, not deferred or hidden.

What changed (SQLite)

• Import invalidates per-system — write_code_system now deletes the re-imported system's rows from concepts_fts, concepts_word_fts, concepts_search_fts and its concepts_fts_built tracker row, in the same transaction as the concept upserts (mirrors the adjacent closure/implicit invalidation). This makes concepts_fts_built the authoritative "FTS current for this system" flag.
• Incremental startup build — prebuild_concepts_fts builds only systems missing from concepts_fts_built (a NOT IN (…) guard on every INSERT — mandatory now the wipe is gone, since FTS5 has no row uniqueness), skips the whole transaction via a cheap up-front gate when nothing is stale, and returns the count built.
• No more blanket wipe — finalize_after_bootstrap and new_inner drop the unconditional DELETE FROM concepts_fts… and only run ANALYZE when a build actually ran (stats persist across restarts).
• CLI hts import ships an index-ready DB — it now finalizes (closures + FTS) at the end of the untimed import, so the server-start finalize is a sub-second no-op. Previously only closures were pre-built here, leaving the full FTS rebuild to block startup.
• Delete cleans up — delete_code_system clears the deleted system's FTS rows + tracker (the FTS5 tables have no FK to cascade from), preventing orphans now that boot no longer wipes.

Net effect: a warm restart, and the benchmark's hts import → serve flow, do zero FTS work before the server binds. Filtered $expand correctness is unchanged (and strengthened — a live $import that changes a display is now reflected without a restart), via the existing lazy ensure_concepts_fts fallback.

All-DB consistency

Postgres needs no code change. It searches a live pg_trgm GIN index on concepts.display (postgres/schema.rs) that the engine maintains transactionally on every write — already incremental, no per-boot rebuild, no tracker. This PR adds a Postgres parity test to pin that invariant as executable.

Not a masking workaround

The banned pattern hides a cost while leaving the work unchanged (e.g. widening the readiness window, which #292 tried and the issue explicitly rejects). This PR does the opposite: it removes the per-boot rebuild and moves the genuinely-once cost onto the once event (import), exactly as closures already do. No sleep, no timeout bump, no backgrounding-past-the-probe. The correctness contract is strengthened, and the new tests fail loudly if a future change reintroduces a per-boot rebuild or drops the invalidation.

Design provenance

Approach vetted by a 5-persona review (SQLite internals, SRE/readiness, performance/benchmark, FHIR-terminology correctness, architecture) — all five converged on this incremental + prebuild-on-import shape and rejected background-after-bind (write-lock contention would pollute the first EX0x benchmark iterations, plus a half-built concepts_search_fts hazard via the non-tracker EXISTS probe).

Tests

• value_set.rs — incremental build is idempotent / no duplicate rows; rebuilds only the invalidated system.
• fhir_bundle.rs — re-import clears the FTS tracker + content (direct Leg A guard).
• terminology_import.rs — end-to-end: re-import changes a display → filtered $expand reflects the new display, stale one gone.
• postgres_http_tests.rs — Postgres parity for the same re-import → filtered $expand behavior.

Verification note

Built and formatted with cargo fmt; the local environment has no C linker so cargo check/test can't link here — relying on CI to compile and run the suite (incl. the Postgres testcontainer parity test). Draft until CI (and ideally a benchmark preflight run on the sqlite leg) is green.

[codecov]

Codecov Report

❌ Patch coverage is 91.16279% with 19 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
crates/hts/src/backends/sqlite/mod.rs	61.11%	7 Missing ⚠️
crates/hts/src/backends/sqlite/value_set.rs	94.11%	6 Missing ⚠️
crates/hts/src/main.rs	0.00%	6 Missing ⚠️

📢 Thoughts on this report? Let us know!