perf: replace window-function retrieval with UNION ALL + per-bank HNSW indexes

[nicoloboschi]

Problem

retrieve_semantic_bm25_combined() used ROW_NUMBER() OVER (PARTITION BY fact_type ORDER BY embedding <=> $1) to get top-N results per fact type. This forced a full sequential scan — pgvector cannot use HNSW indexes when a window function partitions on the ordering column.

Closes #539.

Solution

1. UNION ALL retrieval (retrieval.py)

Replace the window-function CTE with a UNION ALL of per-fact_type subqueries. Each arm has its own ORDER BY embedding <=> $1 LIMIT n, which gives the query planner a shape it can satisfy with an HNSW index scan. Semantic arms over-fetch 5× (min 100) to compensate for HNSW approximation loss; results are trimmed to limit in Python. A single connection/round-trip is preserved (no parallel connections per recall unlike the approach in PR #540).

2. hnsw.ef_search=200 at pool init (memory_engine.py)

Set once per connection via asyncpg.create_pool(init=...) — no per-query SET/RESET overhead. Gracefully ignored if the extension doesn't support it.

3. Per-(bank, fact_type) partial HNSW indexes (bank_utils.py, migrations)

Two root causes prevented the planner from using existing indexes:

• fact_type-only partial indexes: the idx_memory_units_bank_id B-tree always wins when bank_id is in the WHERE clause.
• Global HNSW index (idx_memory_units_embedding): competes with per-fact_type partials for larger partitions (world, observation) and wins.

Fix: create per-(bank_id, fact_type) partial HNSW indexes (both predicates in the WHERE clause) and drop the global index. Index naming uses a stable internal_id UUID UNIQUE column added to the banks table.

Lifecycle:

• Created at bank-creation time (create_bank_hnsw_indexes)
• Dropped on bank deletion (drop_bank_hnsw_indexes, combined with DELETE ... RETURNING to avoid an extra round-trip)

4. Migrations

• a3b4c5d6e7f8: interim fact_type-only partial HNSW indexes
• d5e6f7a8b9c0: adds internal_id UUID UNIQUE to banks, replaces fact_type-only indexes with per-(bank, fact_type) partials, drops global HNSW index, backfills indexes for existing banks

Test plan

• Run cd hindsight-api && uv run pytest tests/ -v — all tests pass
• Run migration on a test DB: uv run hindsight-admin run-db-migration
• Verify with EXPLAIN (ANALYZE, BUFFERS) on a bank with 20K+ rows that the planner selects Index Scan using idx_mu_emb_worl_<uid> (not Seq Scan) for each UNION ALL arm
• Run recall performance benchmark: ./scripts/benchmarks/run-retain-perf.sh and confirm latency improvement vs baseline