[BUG] ThreadSettlementReactor spawns ~50 git/min even when auto-settle is disabled; vcs.refreshStatus sits at 10s p50

[joshdchang]

ThreadSettlementReactor spawns ~50 git/min even when auto-settle is disabled; vcs.refreshStatus sits at 10s p50

Summary

On a self-hosted server with many worktrees, the server spawns git processes
continuously for per-thread bookkeeping. The largest single contributor is
ThreadSettlementReactor.sweep, which performs its full pull-request lookup
before reading the settings that decide whether auto-settlement can happen at
all — so the work runs at full cost even with sidebarAutoSettleOnMerge: false and sidebarAutoSettleAfterDays: null.

The user-visible result is the "Some requests are slow — N requests waiting
longer than 15s" toast and repeated "… is reconnecting" banners, while the host
still shows idle CPU. Measured over 99.5 minutes of trace logs, the two RPCs
named in that toast have a median latency of ~10 seconds.

Environment

t3	0.0.39-nightly.20260904.1278 (also reproduced on …0903.1270, …0902.1257)
Node	v24.19.0
OS	Ubuntu 24.04.4 LTS, 18 cores, 94 GB RAM
Deployment	self-hosted server, remote desktop client
Threads	218 total, 15 unsettled
Worktrees	26, all sharing one 1.6 GB .git (2,465 remote-tracking refs, 57 packs)
Relevant settings	automaticGitFetchInterval: 0, sidebarAutoSettleOnMerge: false, sidebarAutoSettleAfterDays: null

Note the settings row: automatic git fetch is off, and auto-settlement is
fully disabled. This is not the background fetch poller.

Measurements

Parsed from ~/.t3/userdata/logs/server.trace.ndjson*, attributing each
runGitCommand span to its root span by walking parentSpanId.

Window: 99.5 minutes, 11 rotated trace files.

runGitCommand spans: 4838  (49/min)

git spawns by root span:
  2286   ThreadSettlementReactor.sweep
  2103   http.server GET            (the /ws VCS status subscriptions)
   308   processRuntimeEvent
    79   (unknown)
    62   processDomainEvent

Completed-span latency (ms):

span	n	p50	p95	max
ws.rpc.vcs.refreshStatus	49	10,936	31,034	45,366
ws.rpc.review.getDiffPreview	42	9,987	38,493	48,260
ThreadSettlementReactor.sweep	39	15,815	38,459	42,204
runGitCommand	4,838	578	3,036	22,527
sql.execute	77,301	0	7	4,107

sql.execute is included to rule it out: the SQLite layer is not the
bottleneck. The sweep, at a 15.8 s median on a 60 s schedule, occupies
roughly a quarter of wall-clock time by itself.

Direct confirmation that settings do not gate the work. With
sidebarAutoSettleOnMerge: false and sidebarAutoSettleAfterDays: null, over a
180-second window:

git spawns since marker: 376
  190   ThreadSettlementReactor.sweep     <- 63/min, feature disabled
  186   (unknown)

Host-level effect — the box spends about twice as much CPU in the kernel as in
userspace, which is process creation rather than computation:

%Cpu:  19.5 us,  38.5 sy,  33.6 ni,  6.6 id
forks/sec: 19        context-switches/sec: 6868

Sampling the server's direct children for 5 seconds catches a continuous stream
of git processes and their zombies (47 [git] <defunct> in one sample).

V8 CPU profiles of the server's main thread (via --inspect,
Profiler.setSamplingInterval(1000)) during busy periods:

• 15–47% of samples in spawn
• ~18% in spanToTraceRecord / formatTraceExit / flushUnsafe — the
  server's own trace-record formatting
• ~5% in dedupeRemoteBranchesWithLocalMatches (2,465 remote refs)

Root cause

1. The sweep does its expensive work before checking whether it may act

From ThreadSettlementReactor (names as they appear in the published bundle):

const sweep = fn("ThreadSettlementReactor.sweep")(function* (mergedPullRequest) {
  const snapshot = yield* snapshots.getShellSnapshot();
  const candidates = snapshot.threads.filter((thread) =>
    isAutoSettlementCandidate(thread, now) && (...));
  const groups = Map.groupBy(candidates, lookupKey);
  // ...
  const pullRequest = yield* pullRequestFor(group[0]);   // <-- git + `gh pr list`
  yield* forEach(group, (thread) => gen(function* () {
    const settings = yield* settingsService.getSettings; // <-- settings read here
    const settledAt = resolveAutoSettlementAt({
      thread, pullRequest, now: decisionNow,
      autoSettleAfterDays: settings.sidebarAutoSettleAfterDays,
      autoSettleOnMerge: settings.sidebarAutoSettleOnMerge,
    });
    if (settledAt === null) return;                      // <-- always null when disabled

pullRequestFor → GitManager.branchPullRequest fans out into
GitVcsDriver.readConfigValue, resolveDefaultBranchName, remoteExists,
remoteBranchExists, isUnpublishedBranch, resolveRemoteRepositoryContext,
computeAheadCountAgainstBase, and finally gh pr list --head <branch> --state all. That is roughly 6–8 git spawns plus one gh invocation per candidate
group, every 60 seconds (ThreadSettlementReactor.start uses
repeat(spaced("1 minute"))).

When both settings are off, resolveAutoSettlementAt can only ever return
null, so 100% of that work is discarded.

2. Per-thread VCS status refresh scales with open threads

VCS_STATUS_REFRESH_INTERVAL = seconds(30) per subscribed thread, each
refresh running statusDetailsLocal, statusDetailsRemote,
resolveBranchHeadContext, computeAheadCountAgainstBase and friends. This is
the http.server GET bucket above — 2,103 git spawns, 43% of the total. It
correctly pauses when the client goes idle, but with a client open across many
threads it is the second-largest source.

3. Single-threaded amplification

All of this is orchestrated from one Node thread. Once spawn occupies a large
fraction of the event loop, interactive RPCs queue behind it — which is why the
symptom appears while the host still reports 60–70% idle CPU, and why reducing
process priority on other workloads does not resolve it.

What we ruled out

• SQLite — 77,301 sql.execute spans at p50 0 ms, p95 7 ms.
• Background git fetch — automaticGitFetchInterval: 0 throughout; setting
  it produced a large, separately measurable improvement and this is the
  remaining load on top of that.
• Host CPU saturation from other tenants — reproduced with the box at 60–70%
  idle and every competing workload at nice 19.
• GitHub rate limiting — 5000/5000 remaining; gh concurrency is already
  bounded at GITHUB_PROCESS_CONCURRENCY = 4.
• Stale remote refs — git remote prune --dry-run reports 0 prunable.

Suggested fixes

Roughly in order of value-to-effort:

1. Gate the sweep on its own settings. Return early when
   sidebarAutoSettleOnMerge === false && sidebarAutoSettleAfterDays === null,
   or hoist the settings read above pullRequestFor. This alone removes ~47% of
   git spawns for anyone who has auto-settle turned off.
2. Cache pull-request lookups per (repository, branch) with a short TTL.
   The sweep re-resolves identical branches every minute; PR state rarely
   changes that fast, and pullRequests.subscribeMerges already exists for the
   event-driven path.
3. Back off with scale. A fixed 60 s interval means cost grows linearly with
   thread count while the sweep's median duration is already 15.8 s. Consider
   scaling the interval by candidate count, or skipping when the shell snapshot
   is unchanged.
4. Batch the gh calls. One gh pr list --state all per repository, joined
   locally, instead of one invocation per branch.
5. Bound git spawning against the event loop. VCS_PROCESS_CONCURRENCY = 8
   bounds concurrency but not main-thread cost; moving spawn orchestration off
   the main thread would decouple interactive RPCs from bookkeeping.
6. Trace formatting cost. ~18% of main-thread samples in
   spanToTraceRecord/formatTraceExit seems high for always-on telemetry, and
   we could not find a setting or environment variable to reduce or disable it.
   Sampling, or moving formatting off the hot path, would help — as would simply
   documenting a way to turn it down.
7. Expose the interval. A threadSettlementSweepInterval alongside the
   existing backgroundActivity overrides would let large installations tune
   this without a patch.

Reproduction

1. Self-host the server against a repository with a large .git and many
   worktrees (ours: 26 worktrees, 1.6 GB .git, 2,465 remote refs).
2. Keep ~15 threads unsettled and unarchived.
3. Set sidebarAutoSettleOnMerge: false, sidebarAutoSettleAfterDays: null,
   and automaticGitFetchInterval: 0.
4. Watch ps for git processes parented to the server, or parse
   server.trace.ndjson* and attribute runGitCommand spans to their root
   span.

Expected: near-zero settlement-related git activity, since the feature is off.
Actual: ~63 git spawns per minute attributed to ThreadSettlementReactor.sweep.

Happy to supply raw trace files or .cpuprofile captures if useful.

Related issues

Searched before filing; this appears distinct from each of these:

• [Bug]: Background PR status polling via gh pr list drains GitHub GraphQL rate limit #3581 (closed) — Background PR status polling via gh pr list drains
  GitHub GraphQL rate limit. Same gh pr list fanout, but reported as a quota
  problem. This report adds the specific ordering defect (the settings that
  disable the feature are read after the lookup that costs the work) and
  measures the latency and CPU cost rather than quota use. On our install quota
  is not the constraint — 5000/5000 remaining while the symptom occurs.
• Overlapping VCS status refreshes multiply Git and network work #7076 (open) — Overlapping VCS status refreshes multiply Git and network
  work. Covers the explicit vcs.refreshStatus path and in-flight coalescing.
  That corresponds to the http.server GET bucket above (43% of git spawns);
  fixing it would not address the ThreadSettlementReactor.sweep bucket (47%),
  which runs on a timer with no client involved. The two are complementary.
• [Bug]: Local backend becomes CPU-bound and unresponsive during idle VCS/provider refreshes, causing reconnects #4773 (closed) — Local backend becomes CPU-bound and unresponsive during
  idle VCS/provider refreshes. Similar symptom class; this report isolates one
  specific remaining contributor with span-level attribution.

[joshdchang]

Correction to the original report, and a more precise root cause

After reading the actual source rather than the published bundle, one claim in
my report above is wrong and I want to retract it before anyone spends time on
it.

What I got wrong

I claimed that when sidebarAutoSettleOnMerge: false and
sidebarAutoSettleAfterDays: null, "100% of that work is discarded." That is
not correct. In ThreadSettlementPolicy.resolveAutoSettlementAt,
pullRequestSettles returns true when the pull request is closed,
independent of autoSettleOnMerge:

if (pullRequest.state !== "closed" && (pullRequest.state !== "merged" || !autoSettleOnMerge)) {
  return false;
}

So a closed-but-unmerged PR still settles its thread with both settings off, and
the lookup is genuinely needed to discover that. Hoisting the settings read
above pullRequestFor — my suggested fix #1 — would change behavior, not just
performance. Please disregard it.

The measurements in the report stand; only that explanation was wrong.

The actual repeated work

Re-attributing every runGitCommand span to its immediate parent over the
same trace window (5,161 git spawns) points somewhere more specific:

runGitCommand by immediate parent:
  1666   GitVcsDriver.readConfigValue          <- 32% of all git spawns
   583   GitVcsDriver.resolveDefaultBranchName
   338   GitManager.isUnpublishedBranch
   271   GitVcsDriver.remoteExists
   269   GitVcsDriver.resolveBaseBranchForNoUpstream.config
   265   GitVcsDriver.remoteBranchExists
   246   GitManager.branchPullRequest.remotes
   246   GitManager.branchPullRequest.branchRef

span counts:
   246   branchPullRequest
   242   resolvePrLookupRepositoryIdentity
   286   resolveBranchHeadContext
  1056   resolveRemoteRepositoryContext

resolveRemoteRepositoryContext is called 1,056 times, which is exactly
(242 + 286) × 2 — both resolvePrLookupRepositoryIdentity and
resolveBranchHeadContext call it twice, once for the branch remote and once
unconditionally for origin. Each call does a single
git config --get remote.<name>.url and then pure string parsing.

Those 1,056 spawns resolve a very small set of distinct (cwd, remoteName)
pairs — on this install, 26 worktrees that are almost entirely origin. Remote
URLs effectively never change during a session, yet the value is re-shelled for
every branch, on every sweep.

The 60-second PR_LOOKUP_CACHE_TTL does not help here, because
resolvePrLookupRepositoryIdentity runs in branchPullRequest before and
independently of the cached lookup — it has to, since it validates the cached
entry's repository identity. So a PR-cache hit still costs the full
repository-level git fan-out.

Revised suggestion

Memoize resolveRemoteRepositoryContext per (cwd, remoteName) with a short
TTL, in the same style as the existing localStatusResultCache and
prLookupCache in GitManager.ts. On the trace above that removes roughly 20%
of all git spawns on its own, with no behavior change: the function is a pure
read of remote.<name>.url plus parsing, and the existing invalidateStatus
epoch already exists for the cases where a remote genuinely changes.

Suggestions 2–7 in the original report are unaffected. I have a patch for this
one and will open it as a PR shortly.