perf(resolver): keep single-file edits fast during daemon warmup#197
Merged
Conversation
… resolveEdge Preparatory, behaviour-preserving: adds a validateLiveness flag (default off) and two guards in CrossRepoResolver.resolveEdge — skip an edge that is no longer live (edgeStillLive), and refuse a resolution whose target node was evicted (leaving it unresolved). Both are no-ops on the existing whole-pass-locked path, where nothing mutates the graph mid-pass. They become load-bearing for the chunked resolve that releases the resolve mutex between chunks so interactive edits can interleave.
CrossRepoResolver.ResolveAll held the global resolve mutex for its entire pass (~70s on a large warmup), and the same mutex guards an interactive edit's ResolveFileAndIncoming — so any edit landing mid-pass blocked for the full remaining duration. Process the pending set in super-chunks instead: each chunk's compute+apply runs under the lock (atomic, fresh reads), and the mutex is released between chunks (where the pass holds no partial state) so a waiting edit gets in within one chunk. resolveEdge's liveness guards handle any edge/candidate a yielded edit evicted. Default chunk 2048; GORTEX_RESOLVE_CHUNK=0 restores the whole-pass-locked behaviour.
…solve Runs CrossRepoResolver.ResolveAll while an editor goroutine evicts and re-indexes caller files under the same resolve mutex an interactive edit takes — the exact interleaving the chunked path enables. Asserts no panic / race (the half-resurrection an unguarded ReindexEdge on an evicted edge would cause) and that no resolved edge points at a missing node. Passes under -race -count=3.
ReconcileAll (the periodic reconcile janitor) ran RunGlobalGraphPasses UNSCOPED whenever any repo reindexed — re-detecting clones and reseeding the clone index for ALL tracked repos, holding the global resolve mutex for tens of seconds (~52s for 21 repos in one trace). That mutex also guards every interactive edit's resolve, so each janitor cycle that touched even one file stalled concurrent edits for the full pass. Feature B armed the changed-repo scope only on the warmup path; arm it here too so the janitor's clone passes run only for the repos that actually changed this cycle.
The chunked cross-repo resolve ran its liveness guards (edgeStillLive -> GetOutEdges) inside resolveEdge, i.e. once per PENDING edge — O(pending * out-degree). On a warmup pass of ~198k mostly-unresolvable edges that ballooned the pass from ~70s to ~296s and pushed graph-queryable to 377s. Move the guards to the apply: validate only the resolved batch (the few thousand edges that actually changed) before ReindexEdges. Same safety (drop an evicted edge, revert a dead-target resolution) at a fraction of the cost.
The same-repo Resolver.ResolveAll held the global resolve mutex for its whole pass (~24s pre-ready on a large warmup). That mutex also guards every interactive edit's ResolveFileAndIncoming, so a single-file edit during the pre-ready resolve phase stalled tens of seconds — comparable to indexing the whole repo, for one file. Process pending in super-chunks and release the mutex between chunks (compute mutates a clone, so the only live-edge write is the per-chunk apply, which skips an edge a yielded edit evicted). Accumulated jobs feed the post-resolve passes once; guardCrossPackageCallEdges gets the same evicted-edge guard. Validated by TestResolveAll_ConcurrentEdits under -race -count=5.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
A single-file edit (file save /
edit_file) while the daemon is warming up blocked for tens of seconds — comparable to indexing the whole repo, for one file. Measured: editing a heavily-referenced file (internal/config/config.go) during warmup took 44–120 s.Root cause
Interactive edits and the warmup passes share one global resolve mutex (
g.resolveMu): everyResolverbuilt from a graph shares it,ResolveFileAndIncomingtakes it, and three warmup passes held it for their whole duration:So an edit landing mid-pass waited out the full pass — the edit wasn't doing work, it was blocked.
Fix
GORTEX_RESOLVE_CHUNK=0disables,GORTEX_RESOLVE_CHUNK_SIZEtunes). Each chunk's compute+apply is atomic under the lock; only the inter-chunk gap is unlocked, where the pass holds no partial graph state — so a waiting edit gets in within one chunk.guardCrossPackageCallEdgesdrop an edge that is no longer live — validating only the resolved batch, not every pending edge.Result (measured)
Editing
config.goduring warmup: 44–120 s → ~35 ms. Warmup-log timings: master resolve 5.95 s, cross-repo resolve 28.6 s, clone passes scoped to 1 repo (2.2 s vs 52 s for 21).Safety
Two concurrent-edit stress tests (
TestCrossRepoResolveAll_ConcurrentEdits,TestResolveAll_ConcurrentEdits) run eachResolveAllwhile an editor goroutine evicts files under the same mutex — green under-race -count=5. Full build,golangci-lint(0 issues), and the indexer race suite pass.