perf(serve): trigram candidate prefilter to cut O(N) query latency on large graphs (byte-identical results)

[papinto]

Summary

serve's query path scans every node on every query. _score_nodes and
_find_node both loop over the whole graph (for nid in G.nodes(...),
O(nodes x terms)), so query latency scales with total graph size regardless of
where the answer lives. On a large monorepo that is the dominant per-query cost.

This adds a lazily-built, in-memory character-trigram candidate index that
narrows each query to a small candidate superset before the existing scoring
loop runs. Results are byte-identical to today's output, the on-disk format
is unchanged (the index is in-memory only), and a selectivity guard keeps
broad-term queries from ever being slower than the current scan.

This is the query-latency lever discussed under #819 (performance for larger
codebases). It is orthogonal to and complementary with the storage-backend track
in #1297; this one needs no format change and no new dependency.

How it works

• A trigram -> node-ids index is built over exactly the fields the scorer
  already reads: norm_label, the tokenized label_tokens form, nid, and
  source_file (NUL-joined so a trigram cannot span two fields). Because that is
  the complete set of predicates _score_nodes and _find_node match against,
  the index is a complete candidate generator: any node not in the candidate
  set scores 0 under the unchanged scorer, so dropping it changes nothing.
• The scoring and ranking code is untouched. The prefilter is encapsulated
  inside _score_nodes and _find_node; the call sites do not change. IDF stays
  a whole-graph statistic (computed from the full graph, not the candidate
  subset), so scores are identical.
• _find_node candidates are returned in graph-iteration order, so its
  exact, prefix, and substring tie-ordering, and matches[0], are byte-identical
  to a full scan.
• The index is cached on the graph object and auto-invalidated on hot-reload
  (same lifecycle as the existing _idf_cache), so a reloaded graph rebuilds it.

Never-worse guard

Before using the index, a cheap guard (postings-length lookups only, no set
intersection) falls back to the existing full scan when the index would not help:

• a query term too short to form a 3-char trigram, or
• the term's rarest trigram still appears in more than ~10% of nodes (a broad
  term like model or client whose candidate set would approach the whole
  graph).

So narrow, specific queries take the fast path; broad queries run the unchanged
scan at ~1.0x (never slower). The fast path is selected automatically per query.

Eager build at startup

The index builds on first use. Since a serve process exists to answer queries,
it is warmed right after load, and a reloaded graph is warmed before it is
swapped in, so neither the first query nor the first post-reload query pays the
one-time build cost (~2.3 s at 54k nodes, ~7.3 s at 177k).

Measured (our graphs; verified byte-identical against the current scorer)

End-to-end on the integrated path (the prefilter monkeypatched off gives today's
baseline), best-of-3 per query:

Graph	Nodes	Index memory	Build time	End-to-end query speedup (median)	Fidelity
A	~54k	~20.5 MB	~2.3 s	150.9x (fast path on 42/47)	47/47 score + 47/47 find_node identical
B	~177k	~63.7 MB	~7.3 s	159.0x (fast path on 41/50)	50/50 score + 50/50 find_node identical

• Fidelity: every query and every _find_node lookup is byte-identical to
  the current path across both graphs. New test_serve.py coverage is included.
• Memory surcharge is ~5-9% on top of the already-resident graph, and the
  label_tokens portion shrinks with scale (~4.7% at 54k -> ~3.6% at 177k)
  because almost every label token is already a substring of norm_label. Build
  cost is roughly 41 ms per 1k nodes.
• Broad or common-term queries hit the guard and run the existing scan unchanged
  (~1.0x; the slowest observed cases are within best-of-3 timing noise on a
  multi-second scan, since the fallback adds only a cheap O(query-trigrams) guard
  probe). The relative win is concentrated where it matters (specific lookups on
  big graphs) with no regression on the tail.

Scope and non-goals

• In-memory only. No storage-format change, no new dependency, no CLI change.
• Not a throughput or concurrency change. It shrinks per-query work, but the scan
  was already GIL-bound, so scale-by-process is unchanged.
• The storage-backed approach (instant load, shared memory across replicas,
  incremental update) is the separate track in RFC: optional SQLite tiered storage backend for serve (~17–64× lower query latency, byte-identical results) #1297. This is the
  no-format-change query-latency win that stands alone.

[safishamsi]

Merged into v8 as 53edd27 (cherry-picked). I independently verified the correctness-critical claim by running the PR's own fast == full equality tests (8 prefilter/trigram tests) plus the full serve+explain suite (65 tests) — the prefilter is a true superset of the exhaustive scorer, ranking/ordering unchanged, with short/CJK/low-selectivity fallback. Full suite green on 3.10+3.12. Thanks @papinto!