HTS sqlite server startup rebuilds concepts_search_fts on every boot — exceeds 300s with full terminology (debug), breaking the sqlite benchmark leg

[mauripunzueta]

Summary

The hts sqlite server no longer becomes ready with a full terminology corpus loaded: startup silently rebuilds the FTS index (including the concepts_search_fts table) over all concepts on every boot, which in a debug build exceeds even a 300s readiness window. The server logs backend init, then goes silent — it never binds/serves. This breaks the sqlite leg of the HTS Terminology Benchmark workflow (Start HTS server step fails because /health never returns 200). Postgres is unaffected.

Not a regression in any observability/benchmark PR

Reproduced on plain main (control run 29603533041, backend=sqlite, tests=preflight-only): the sqlite leg fails at Start HTS server with the identical log signature. The post-#233 baseline run 29023969774 had sqlite: success — so this appeared between that baseline and current main.

Root cause

crates/hts/src/main.rs (sqlite run_server) calls finalize_after_bootstrap() unconditionally after bootstrap sync, before binding:

bootstrap_sync_sqlite(&backend, &config).await?;
backend.finalize_after_bootstrap()?;   // <-- runs on every startup
...
let listener = TcpListener::bind(&addr).await?;   // never reached

finalize_after_bootstrap() (crates/hts/src/backends/sqlite/mod.rs:448) deletes and rebuilds all FTS content, then prebuild_concepts_fts:

DELETE FROM concepts_fts; DELETE FROM concepts_fts_built;
DELETE FROM concepts_word_fts; DELETE FROM concepts_search_fts;   // concepts_search_fts added by #273
...
value_set::prebuild_concepts_fts(&conn);   // repopulates over ALL concepts (full SNOMED)

The concepts_search_fts table was introduced by 39626948a (PR #273, "fix(hts): create concepts_search_fts and make its ranked query valid"). Its addition pushed the per-boot FTS rebuild over full SNOMED past the readiness window in a debug build. Before #273 the table didn't exist, so the rebuild was cheaper and startup fit within 60s.

Evidence

• Failing sqlite server log (both PR fix(observability): stop paying for unconsumed per-request instrumentation (#227) #292 runs and the main control) shows only:

  INFO hts: Starting Helios Terminology Server ... storage_backend=sqlite
  INFO helios_hts::backends::sqlite: SQLite terminology backend initialized ...
  

  then silence — no "HTS listening", no bind. Confirms it is stuck in finalize_after_bootstrap, not a crash (no panic in the log).
• Postgres starts in ~0.4s (no equivalent per-boot FTS materialization) and its benchmark legs pass.
• main control run: 29603533041 (sqlite, Start HTS server = failure).

Impact

• The sqlite HTS benchmark leg cannot run on current main — issue HTS Terminology Server perf regression on main since v0.2.0 (EX02/EX08 expansion collapse + ~30% uniform drop) #227's sqlite numbers are currently unreproducible.
• Any sqlite HTS deployment with a large corpus pays a full FTS rebuild on every restart, and may fail health checks with a tight readiness probe.

Suggested directions

• Make the startup FTS rebuild conditional — skip when the FTS content is already present/current (mirror the idempotent migrate_concept_closure "only rebuild what's missing" pattern), rather than unconditional DELETE+repopulate on every boot.
• And/or gate it so it does not block /health/bind (build in the background, or serve readiness once bound), and/or make it materially faster for concepts_search_fts.
• The benchmark workflow already widened the readiness window to 300s (PR fix(observability): stop paying for unconsumed per-request instrumentation (#227) #292); that is insufficient here and is not the right fix — the startup cost itself needs to come down.

Reproduction

Run the HTS Terminology Benchmark workflow on main with backend=sqlite, tests=preflight-only. The Start HTS server step fails after the readiness window.