Server: concurrent cold location builds serialize (p99 3.5s, 16.2s observed)

[shuv1337]

Summary

Cold location-services builds serialize against each other. Concurrent builds for different directories all complete at the same instant after a shared delay, rather than proceeding independently. In production logs this reaches 16.2s, and it sits directly on the HTTP response path because LocationMiddleware builds the layer inline.

Not root-caused. Filing with the evidence gathered while diagnosing the TUI catalog delay (#351), which is a separate defect.

Evidence

Duration distribution

512 location services booted samples from ~/.local/share/opencode/log/opencode.log:

percentile	duration
p50	62 ms
p90	275 ms
p95	576 ms
p99	3521 ms
max	16205 ms

Slowest ten: shuv2code 16205ms, cloudflare-os 16198ms, shuvbot 16198ms, shuv2code 4197ms, shuvbotta 3956ms, shuvbotta 3537ms, shuvbot 3521ms, shuv2code 2792ms, project-radar 2789ms, handy 1884ms.

The 16.2s incident

Three different directories, same server process (run=b31711eb), completing within 20ms of each other after a 16.2s stall:

2026-08-05T20:00:58.474Z location services booted directory=/home/shuv/repos/cloudflare-os durationMs=16198
2026-08-05T20:00:58.485Z location services booted directory=/home/shuv/repos/shuv2code   durationMs=16205
2026-08-05T20:00:58.493Z location services booted directory=/home/shuv/repos/shuvbot     durationMs=16198

Immediately before them, the global-config watcher subscriptions for all three carry the same span:

2026-08-05T20:00:58.425Z watcher subscribe path=/home/shuv/.config/opencode http.span=16149
2026-08-05T20:00:58.426Z watcher subscribe path=/home/shuv/.opencode        http.span=16150
2026-08-05T20:00:58.433Z watcher subscribe path=/home/shuv/.config/opencode http.span=16153
2026-08-05T20:00:58.433Z watcher subscribe path=/home/shuv/.opencode        http.span=16153
2026-08-05T20:00:58.438Z watcher subscribe path=/home/shuv/.config/opencode http.span=16143
2026-08-05T20:00:58.438Z watcher subscribe path=/home/shuv/.opencode        http.span=16143

Plugin activation churned throughout the window (agent.updated / command.updated / catalog.updated at 20:00:47.093, 47.105, 52.859, 53.075, 54.621, 54.632), suggesting repeated catalog reloads rather than a single blocking call.

Reproduction

Six fresh git repos with an opencode.json, hitting /api/provider against a warm running server:

sequential cold builds:  31 ms, 31 ms, 35 ms
concurrent cold builds: 322 ms, 331 ms, 338 ms   <- all finish within 16 ms of each other

A 10x slowdown purely from concurrency, with the completion-clustering signature. The absolute cost here is small because these are trivial directories; the constant factor scales with real repositories and real config.

Why it matters

packages/server/src/location.ts:56 provides locations.get(ref) inline on the request, so the whole build is on the HTTP response path. LayerMap (packages/core/src/location-services.ts:118-141) memoizes per canonical Location.Ref with idleTimeToLive: "60 minutes", so any directory not touched in an hour pays full cost — and multiple clients or tabs opening different projects at once is the exact case that serializes.

Candidate causes, unverified

• LayerMap build serialization (packages/core/src/location-services.ts:118-141) — ~50 location-scoped nodes constructed per entry; the completion-clustering points here or at something it awaits.
• Config.discover() (packages/core/src/config.ts:377) — walks to the filesystem root reading and decoding opencode.json(c) at every level.
• loadWellknown() (packages/core/src/config.ts:372 → packages/core/src/wellknown.ts:63, 87-93) — issues an HTTP GET per registered origin with no timeout, and is reached inside the location layer build, so it is on the response path. Zero cost when the wellknown:sources KV is empty, which is the common case.
• Project.resolve() (packages/core/src/project.ts:244-269) — git repo discovery, git remote, git worktree list, DB persist.
• Sequential plugin activation (packages/core/src/plugin.ts:87-107) — plugins load in a for loop inside State.batch, so one slow plugin stalls the generation; models-dev, opencode, anthropic and openai each block the next.
• State.reload debounce (packages/core/src/state.ts:37, 112-114) — reloadDebounce = 500 delays every catalog.reload() by at least half a second.

Suggested next step

Add timing spans inside the location layer build so location services booted attributes its duration to a phase (config discovery, wellknown, project resolve, node construction) rather than reporting one opaque total. The existing log line already carries durationMs; a breakdown would immediately settle which of the above is responsible, and whether the 16.2s is a shared lock or an unbounded network await.