perf: CLI cold start is 21s+ — performance sprint needed

[laynepenney]

Problem

synapt recall search takes 21-27 seconds from the CLI, even for simple queries. This makes the CLI unusable for demos, scripts, and quick lookups. The MCP server doesn't have this problem because it stays warm.

Breakdown (48,700 chunks, 263MB recall.db)

Phase	Time	%
Python startup + imports	~2s	9%
Embedding model load (all-MiniLM-L6-v2, 22M params)	~12s	55%
SQLite chunk loading (48K rows → Python objects)	~5s	23%
Actual search (FTS5 + scoring + formatting)	~2s	9%
Output rendering	~0.5s	2%

The search itself is fast. The 21 seconds is all overhead.

Proposed Performance Sprint

P0 — Immediate impact

Lazy chunk loading
Don't load all 48K chunks into memory. FTS5 returns matching rowids — only materialize the chunks that match. This alone could drop the 5s chunk loading to <100ms for typical queries.

Embedding server / persistent process
The MCP server already keeps the model warm. For CLI use, either:

• Ship a tiny embedding daemon that starts on first use and stays alive
• Use ONNX runtime (3-5x faster model load)
• Cache embeddings in SQLite so the model is only needed for new/unseen queries

Suppress HuggingFace HTTP noise
Every CLI invocation makes 20+ HTTP HEAD requests to check model freshness. Set HF_HUB_OFFLINE=1 after first download, or use TRANSFORMERS_OFFLINE=1.

P1 — Structural improvements

Index sharding by time
48K chunks in one DB is fine for search but slow for full-table scans. Shard by month/quarter so only recent shards are loaded by default. Older shards searched on demand.

Precomputed search cache
For the MCP recall_quick path: cache the top-N results for common query patterns (status queries, file lookups). Invalidate on new session ingest.

Profile-guided optimization
Add --profile flag to CLI that outputs timing breakdown per phase. Use this to catch regressions.

P2 — Future-proofing

Performance regression tests
CI tests that assert:

• CLI search completes in <5s for indexes up to 50K chunks
• MCP recall_quick responds in <500ms
• Index build completes in <60s for 50K chunks
• Embedding model loads in <3s (ONNX) or is pre-warmed

Benchmark tracking
Track CLI response time across versions in evaluation/ — same pattern as LOCOMO benchmarks but for latency. Plot over time, alert on regression.

Scaling targets

Chunks	CLI target	MCP target
10K	<3s	<200ms
50K	<5s	<500ms
200K	<10s	<1s
1M	<15s	<2s

Context

Discovered during demo recording (2026-04-02). A 21-second recall search makes the "aha moment" demo impossible — viewers won't wait. The MCP path is fine for agent use, but the CLI is the first thing a new user tries after pip install synapt.

First impressions matter. A 21-second CLI search kills adoption.

Related

• Demo recording: demo/aha-moment.tape
• Embedding model: sentence-transformers/all-MiniLM-L6-v2 (22M params, ~80MB on disk)
• Recall DB: .synapt/recall/index/recall.db (263MB for 48K chunks)

🤖 Generated with Claude Code

[laynepenney]

Update: Sharding + Lazy Loading must ship together

Current state

ShardedRecallDB exists but is Phase 1 only — it splits files into data_001.db...data_005.db but load_chunks() still iterates ALL shards and loads everything into memory (line 112-119 of sharded_db.py).

Phase 2 (dynamic shard attach/detach, FTS query routing) was designed but never built. The docstring says:

Phase 2 (follow-up): routes chunk FTS queries to per-quarter data shards, merges results, and keeps index.db for knowledge/clusters.

Why splitting alone doesn't help

Running synapt recall split today creates 5 shards of 10K chunks each. But load_chunks() loads all 5 sequentially — same 48K chunks, same memory, same 21-second cold start. No speed improvement.

Revised P0 plan

Ship split + lazy loading as one unit:

1. FTS routing through index.db — index.db gets a chunk-to-shard mapping table. FTS5 query hits index.db, returns matching chunk IDs + shard locations.
2. Dynamic ATTACH — only ATTACH DATABASE the shard(s) containing matched chunks. A query matching 5 chunks in shard 2 never touches shards 1, 3, 4, 5.
3. Split command — synapt recall split remains as-is, but now the lazy loader makes it actually fast.

Expected impact: 48K chunks → only load ~10-50 matching chunks → sub-second search even from cold CLI.

Don't split production DB until Phase 2 ships

Per Layne: risk of debugging sharding issues without the lazy loading payoff isn't worth it. Split + lazy load ship together or not at all.

[laynepenney]

Additional: default to sharded layout for new installs

Per Layne: users shouldn't have to run synapt recall split manually. New indexes should use the sharded layout by default.

Change

• synapt recall build creates index.db + data_001.db from the start (not recall.db)
• synapt recall split becomes a migration command for existing monolithic DBs only
• ShardedRecallDB.open() already auto-detects both layouts, so this is backwards-compatible

Migration path

• Existing users: synapt recall split (one-time, after lazy loading ships)
• New users: sharded by default, no action needed

[laynepenney]

Follow-up correctness blocker after #448: sharded search still needs shard-qualified row ids. Opening a separate bug so the performance sprint tracks FTS population (#448) separately from merged rowid correctness. Until that lands, monolithic remains the safe correctness mode for CLI search.

[laynepenney]

Progress update (Sprint 13)

P0 — Complete

• Lazy embedding loading + numpy storage: PR perf: lazy embedding loading + numpy-backed storage (recall#435) #605 (merged). Defers 48K x 384 embedding load from init to first search. Numpy storage cuts memory from ~516MB to ~74MB.
• HF HTTP suppression: Already implemented (verified during investigation).
• --profile flag: PR feat: search --profile flag for timing diagnostics #606 (merged). Shows per-phase timing breakdown for ad-hoc debugging.

P1 — Partial

• Benchmark command: PR feat: benchmark command for search pipeline metrics #607 (merged, closes Benchmark: baseline performance metrics for search pipeline #505). synapt recall benchmark reports cold/warm start, per-query p50/p95/p99 latencies, memory footprint. Supports --json for CI tracking.
• Intent routing optimization: PR feat: code + project intent routing for recall_quick #608 (merged, closes recall_quick: improve intent routing for project-level vs code-level queries #412). Code and project queries now route to appropriate depth/boost settings, reducing irrelevant results.
• Index sharding by time: not started
• Precomputed search cache: not started

Remaining bottleneck

The 12s embedding model load (55% of cold start) is still the biggest factor. With lazy loading it's deferred to first search, but the first search still takes ~12s for model initialization. Options:

1. ONNX runtime for faster model load
2. Embedding daemon (warm process)
3. Pre-embed queries server-side (for MCP, already warm)

[laynepenney]

A schema-drift defect that sat underneath this issue has been fixed on dev, and the measured effect is worth recording here even though it does not resolve the 21s+ cold start this issue reports.

The defect. Three FTS schema detectors compared the CREATE VIRTUAL TABLE statement stored in sqlite_master against the module constant for that table. SQLite stores a CREATE statement as executed and without the trailing statement terminator, while each constant is written as a runnable statement ending in );. The two could therefore never compare equal, so all three detectors reported drift on every open and every FTS table was dropped and rebuilt each time a store was opened. Whitespace normalisation alone does not close it: strip() removes the newline after the semicolon and leaves the semicolon. Both sides of every comparison now go through one normalising helper.

Method. Two worktrees, one at the commit before the fix and one at the commit after. Runs interleaved rather than blocked, because a blocked design confounds the arm with thermal and load drift on the measuring machine. A fresh copy of the store for every run, because the pre-fix path writes — a reused store would let the second run inherit the first run's mutations. One interpreter throughout, with the import path pinned per arm and the resolved module path printed, so neither arm can silently run a machine-wide install instead of the code under test. Three repetitions per arm.

Result, synapt stats: median wall-clock fell by 46%.

The load-bearing evidence is not the stopwatch. Each run hashed the store before and after. The pre-fix arm mutated the store on every single run — that mutation is the rebuild — and the post-fix arm left it byte-identical on every single run. Three for three in both directions. A timing delta alone would be much weaker; this says directly that the rebuild happened before and does not happen now.

Scope, stated because it is easy to misread as a null result. synapt resume shows no delta. That is not "the fix had no effect there": in the resume runs the pre-fix arm also left the store unchanged, and the pre-fix path rebuilds unconditionally whenever it opens the store, so an unchanged store proves resume never opens the index at all. The fix reaches commands that open the index; resume is not one of them.

What this does not claim. The cold start reported in this issue is dominated by model load and chunk materialization, neither of which the measured command pays. This removes a real, repeatable, per-open cost from every command that opens the index. It is not the 21s.