2.0: managed-service restart causes reconnect herd and resource spikes

[kitlangton]

Summary

Opening a V2 TUI during an automatic update caused the shared managed service to be replaced, disconnected every existing TUI, cold-booted many location graphs concurrently, delayed SSE readiness, and left the new TUI slow and partially rendered. A later explicit service restart reproduced the reconnect herd and pushed location startup to 19 seconds.

This issue collects the coupled V2 managed-service lifecycle concerns exposed by the incident. Confirmed observations are separated from hypotheses; the current logs do not identify the dominant location-startup or memory owner.

Environment

• opencode version during incident: 0.0.0-next-15293 updating to 0.0.0-next-15295
• opencode version during follow-up capture: 0.0.0-next-15296
• OS: macOS 26.5.1, Darwin 25.5.0, arm64
• Terminal: WarpTerminal, TERM=xterm-256color, COLORTERM=truecolor
• Shell: /bin/zsh
• Install/channel: global Bun install, next
• Active plugins: @warp-dot-dev/opencode-warp, ~/.config/opencode/plugins/share-with-team.ts, and the repository debug workspace plugin
• Shared V2 database during follow-up: 861 MB, 223,529 durable events, 589 sessions, 19,185 projected session messages
• Shared text log during follow-up: 472 MB

Reproduction

Automatic-update incident

1. Keep several V2 TUIs connected to the managed background service from different directories.
2. Open another opencode2 TUI while a newer next version is available.
3. Allow the default automatic update policy to install the newer global CLI package.
4. Start a client from the newly installed version.

Explicit restart control

1. Keep seven V2 TUIs connected from several directories.
2. Run opencode2 service restart.
3. Observe reconnect attempts, service election, and location startup timings.

The automatic-update incident was observed once. The explicit restart reproduced the reconnect and cold-location herd.

Expected Behavior

• Updating the globally installed CLI should not unexpectedly replace a shared service and interrupt all connected clients without coordinated UX.
• A service replacement should expose one coherent updating/restarting state to connected TUIs.
• Reconnecting clients should not create an unbounded cold-start herd across every recently active location.
• Event-stream connection timing should distinguish a live-but-busy server from an unavailable server.
• Startup and resource diagnostics should identify the expensive subsystem rather than only reporting one aggregate location duration.

Actual Behavior

• A newly opened next-15293 TUI installed next-15295 in the background.
• A subsequent next-15295 client required an exact-version service and replaced the healthy next-15293 service.
• Six existing TUIs disconnected together during the automatic-update incident.
• The replacement server cold-booted at least eight location graphs concurrently. The affected location took 11.97 seconds; other locations took up to 15.35 seconds.
• The affected TUI's first SSE attempt timed out after two seconds, then connected on retry about 4.5 seconds later.
• The reported session was created about 51 seconds after the original TUI launch and initially rendered as a partial home screen.
• An explicit restart later disconnected seven TUIs. All seven reached reconnect attempt 4 and launched serve --service election attempts. One server won, then ten location graphs booted concurrently, taking up to 19.10 seconds.
• Every connected TUI also writes every durable SSE event to the shared text log, multiplying identical records by client count. This is tracked separately in V2: connected TUIs duplicate durable event logs #36283.
• CPU and memory remained high after startup. The available samples identify broad Bun/WebKit allocator and graphics regions but do not localize the retaining code path.

Additional Context

1. Automatic update and exact-version replacement

The default TUI forks an updater check before resolving the server:

• packages/cli/src/commands/handlers/default.ts

The updater installs the global package in place. Both true and "notify" currently select the upgrade action, and the managed server also runs this updater every ten minutes:

• packages/cli/src/services/updater.ts
• packages/cli/src/server-process.ts

Managed server resolution defaults to exact-version replacement. Service.start() kills a healthy mismatched service before spawning the replacement:

• packages/cli/src/services/server.ts
• packages/client/src/effect/service.ts

Incident timeline, UTC:

15:46:26.110 TUI starts on next-15293
15:46:27.115 update check: current=15293 latest=15295
15:46:30.339 global Bun install updated to 15295
15:46:42.566 new TUI starts on 15295
15:46:42.943 six existing event streams disconnect
15:46:44.470 replacement serve --service starts

There is update-confirmation UI in the TUI for installation.update-available, but the CLI updater path above directly installs the package. Repository search found the event schema and consumer but no publisher in the V2 package set.

2. Cold-location herd

Automatic-update replacement timings:

/private/tmp                                                    5,029 ms
/Users/kit                                                      5,282 ms
/Users/kit/code/open-source                                     7,188 ms
.../opencode/packages/opencode-test-plugins                    11,974 ms
/Users/kit/code/projects/Hex                                   12,372 ms
/Users/kit/code/open-source/effect-smol                        12,829 ms
/Users/kit/code/open-source/openeval                           15,329 ms
/Users/kit/code/open-source/browser-control                    15,346 ms

Explicit restart at 16:09 UTC was a stronger control. Seven clients disconnected, all reached reconnect attempt 4, and seven serve --service contenders launched. Election produced one server (run=0349cbfd). Its ten concurrent location boots included:

/Users/kit                                                      2,529 ms
/Users/kit/code/open-source                                     3,418 ms
/Users/kit/code/projects/Hex                                   11,006 ms
/Users/kit/code/open-source/browser-control                    12,947 ms
/Users/kit/code/open-source/opencode                           15,731 ms
.../opencode/packages/opencode                                18,443 ms
.../opencode/packages/opencode-test-plugins                   18,723 ms
/Users/kit/code/open-source/effect-smol                        19,097 ms

location services booted is one aggregate timer around acquisition of the full location layer. It does not attribute time to configuration discovery, plugins, MCP, search/indexing, watchers, or scheduler contention.

3. Two-second SSE first-frame timeout

The TUI gives api.event.subscribe() two seconds to yield its first event, expected to be server.connected:

• packages/tui/src/context/sdk.tsx
• packages/tui/src/context/sdk.tsx

Affected TUI timeline:

15:46:56.937 event stream connecting, attempt 0
15:46:58.938 disconnected: Timed out connecting to server
15:47:01.910 event stream connecting, attempt 1
15:47:03.470 event stream connected

The server was alive during this period and was concurrently completing location boots. The evidence shows first-frame latency under load; it does not yet show whether the delay occurred in HTTP acceptance, SSE setup, event-loop scheduling, or client processing.

4. Duplicate event log amplification

Each TUI logs durable events after decoding them from the shared SSE stream. During the incident:

15:46 client durable-event lines: 2,842
15:47 client durable-event lines: 3,252
15:48 client durable-event lines: 9,814

Identical aggregate IDs and sequence numbers appeared under seven client run IDs. See #36283 for the narrow logging-semantics issue.

5. CPU and memory evidence

At 12:44 local time, ps reported:

managed server: 74% CPU, 2,999,472 KiB RSS
oldest TUI:     38% CPU, 2,193,824 KiB RSS
other TUIs:     20-33% CPU, approximately 1.0-1.86 GiB RSS

RSS includes resident shared mappings and should not be summed as unique physical memory. vmmap -summary provided a more conservative measurement:

managed server physical footprint: 999.5 MB, peak 1.8 GB
sample TUI physical footprint:      577.5 MB, peak 861.6 MB

Large resident regions in the server snapshot included approximately 1.3 GB WebKit Malloc, 382 MB MALLOC_SMALL, 373 MB MALLOC_LARGE, 359 MB IOAccelerator, and 240 MB JS VM Gigacage. The sampled TUI included approximately 1.2 GB WebKit Malloc, 437 MB JS VM Gigacage, 270 MB IOAccelerator, and 85 MB VM_ALLOCATE resident mappings.

vmmap warned that WebKit malloc-zone analysis was incomplete. The three-second stack samples were mostly unsymbolicated Bun/JSC frames and do not identify the retaining subsystem. No claim is made yet that event handling, OpenTUI, indexing, plugins, or another subsystem is the primary memory cause.

6. Observability gaps

• No per-node location startup timings.
• No explicit service lifecycle event for update pending, restart scheduled, draining, replacing, or ready.
• No measurement separating TCP/HTTP connection, first SSE byte, and first decoded event.
• No heap snapshot or allocation profile tied to server/TUI growth.
• The shared text log reached 472 MB; rotation behavior was not visible in the incident evidence.

Open design questions

• Which process owns update discovery and installation: each TUI, the managed service, or a separate launcher?
• Should automatic update download/stage separately from activation?
• What explicit user action or safe boundary permits replacement of a shared service with active clients and executions?
• What state should connected TUIs render while an update is available, staged, restarting, reconnecting, or ready?
• Should reconnecting clients lazily restore only their current location rather than eagerly recreating every recently active location graph?
• Should SSE connection timeout measure transport establishment separately from first application event?
• What is the intended text-log policy for dense durable events already stored in SQLite?
• Which memory/CPU instrumentation can attribute retained resources to location services, session projections, event reducers, OpenTUI, plugins, or Bun/JSC?

[kitlangton]

Per-node acquisition profiling identified eager Snapshot repository discovery as the largest measured location-readiness cost. In one instrumented concurrent cold start, Snapshot accounted for 552-2,301 ms per location and dominated each critical path.

PR #36290 defers and memoizes that work until the first snapshot operation. Bracketed seven-run A/B measurements reduced median single-location readiness from 426-496 ms to 137 ms, and the slowest concurrent location from 1,656-1,987 ms to 919 ms.

The next measured costs are concurrent project resolution (about 600 ms per location in the sampled herd), config discovery (up to 1,050 ms), and eager Shell output setup (up to 635 ms). Those need separate benchmarked changes rather than being folded into the Snapshot PR.

[kitlangton]

Follow-up experiments after #36290:

• Keep: combine Git repository discovery from three sequential git rev-parse subprocesses into one. Across five concurrent repositories, 20 alternating runs improved the median from 23.3 ms to 12.2 ms (-48%). Existing Git and linked-worktree tests pass. The commit is on git-discovery, stacked behind fix(core): defer snapshot repository discovery #36290.
• Discard: parallelize config target existence checks. The exact operation measured 0.093 ms sequential versus 0.037 ms concurrent, far below the observed startup noise and not worth changing precedence-sensitive code.
• Discard: cap concurrent location builds. In an isolated four-location harness, unbounded acquisition completed in 155 ms median; limit 2 took 204 ms (+32%), and limit 1 took 273 ms (+76%). Once eager Snapshot work is removed, parallel acquisition is beneficial.

The next investigation should avoid broad location-level guesses and target a measured operation directly. Current evidence no longer supports config scanning or build concurrency as primary bottlenecks.

[kitlangton]

The cold-location investigation produced two merged improvements:

• fix(core): defer snapshot repository discovery #36290 removed eager Snapshot repository setup from Location acquisition. Bracketed medians improved from roughly 463 ms to 134 ms for one cold location and 1,954 ms to 707 ms for the slowest concurrent location.
• refactor(core): combine git discovery queries #36321 reduced subsequent Git discovery from three subprocesses to one. The final alternating benchmark improved the five-repository batch from 33.2 ms to 11.5 ms.

Config existence checks and Location build concurrency caps were measured and discarded. Keeping this issue open because coordinated update/replacement UX, SSE readiness under load, and the reported CPU/memory behavior remain unresolved.