fix(python): batch rooted usage-graph existence checks - #5
Merged
sontek merged 2 commits intoSep 4, 2026
Conversation
Python's whole-workspace ("rooted") usage-graph scan resolved every
candidate reference with a live, unbatched SQLite query via
IAnalyzer::definitions(), one round trip per reference. On a large
Python monorepo this took 25+ minutes; the largest Go monorepo
measured with an already-batched equivalent path takes under 10.
record()'s existence check, and two sibling methods with the same
shape, now defer their check into a pending buffer in rooted mode
only (bounded mode, already a cheap in-memory hash check, is
unaffected). After each file's walk, prefetch_definitions batches
every deferred name into a couple of store round trips before the
checks are resolved for real, reusing the exact same underlying
lookup and cache other languages already use via the same trait
method.
build_python_edges also opened its relational frontier fresh per
file instead of once for the whole parallel scan, losing cross-file
batching for the (smaller) receiver-type-resolution path too. It now
opens once around the whole fan-out, matching Java's existing
build_java_edges/with_java_graph_source placement.
…p accept/record logic record_direct_or_namespace_fallback's fallback loop called edges.record_kind directly instead of routing back through the accepts_target check, so in bounded (targeted) scans a namespace re-export candidate outside the caller's requested target set could still get recorded as an edge, since record_kind's own guard only checks workspace membership, not target-set membership. Added a regression test that reproduces it with a re-exported symbol whose direct fqn doesn't resolve, confirmed it fails on the prior commit, and fixed it by routing every fallback candidate through the same accept-and-record helper the direct path already uses. Also extracted record_direct/record_namespace_fallback/ record_ancestor_fallback, shared between each pending variant's bounded-mode immediate branch and its resolve_pending replay, so the two copies of each variant's accept/record logic can't drift apart again the way this bug just showed they can.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Python's whole-workspace usage-graph scan resolved every candidate reference with a live, unbatched relational-store query, one round trip per reference. On a large real-world Python repo this made the scan take tens of minutes. Every other language already batches this same kind of lookup. Python's scan just never routed through it.
recordand two sibling methods now defer their existence check into a buffer, in whole-workspace mode only. Targeted scans already resolve in-memory and stay untouched. After each file's walk, one batched call warms the same cache the point lookup already reads. The deferred checks then resolve against it.Measured directly against a real Python repo checkout, isolating the changed code from the unrelated one-time indexing cost: 28.2 minutes before, 12.6 minutes after. Same edge count both times (226,484), confirming a pure speedup with no resolution difference.
Also fixes a correctness bug the review surfaced. A namespace-import candidate could get recorded as an edge in a targeted scan even when it was not one of the caller's requested targets. The fallback path recorded it directly instead of routing back through the target-set check. Added a regression test that reproduces it and confirmed it fails on the prior commit.
Two smaller costs remain, called out for a follow-up rather than folded into this fix. An ancestor-hierarchy lookup now runs unconditionally on a common path instead of only after a failed direct match, though it is bounded by distinct class count, not reference count, and already memoized per class. The new batching is also per file rather than across the whole workspace, unlike this codebase's own Rust and Java equivalents. Both are real but small next to the fix above.