Add a memory-pressure guard to executor spawns (warn by default)#681
Merged
Conversation
Nothing bounded how many agent sessions could be live at once. Each session is a claude process plus its MCP servers, so a queue that fans out freely can walk a workstation into swap thrashing — at which point every session, dev server and test run on the box gets slower, including the ones already doing useful work. Observed on a 24GB machine: 36 live sessions, 393 processes, of which 35 belonged to tasks already done or blocked. Deliberately NOT a concurrency cap. A fixed "max N sessions" is a hidden wall: the right number depends entirely on the machine and what else is running, and someone who legitimately wants dozens of concurrent agents shouldn't be told no by an arbitrary constant. This gates on actual memory pressure instead — with headroom, spawn as many as you like; the guard only has an opinion once the kernel says memory is genuinely short. The signal is Darwin's kern.memorystatus_level, the same free-memory percentage Activity Monitor's pressure graph derives from. Its zones are green 100-50, yellow 50-30, red 30-0, so the default threshold of 20 fires only well into the red. Where the signal is unavailable the guard is inert. Default mode is "warn": log loudly, never block. Behaviour is unchanged for everyone unless they opt in, and a test pins that property specifically. TY_MEMORY_GUARD=off|warn|block (default: warn) TY_MEMORY_GUARD_MIN_FREE_PCT=<0-100> (default: 20) In block mode the spawn returns ErrMemoryPressure carrying the actual numbers and the override, so if it ever does bite it says why and how to get past it rather than silently stalling. Wired into both spawn choke points: createTmuxWindow (daemon) and EnsureTaskWindow (TUI/API). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The guard was Darwin-only: on Linux the kern.memorystatus_level sysctl fails,
systemFreeMemoryPct reports "unknown" and the guard silently does nothing. That
is the safe failure mode, but it left the agent-server fleet — which is where
an unattended fan-out is least likely to be noticed — with no protection at all.
systemFreeMemoryPct is now per-platform behind a stable contract ("percent of
memory still available, or ok=false"), so the threshold means the same thing
everywhere:
darwin kern.memorystatus_level (what Activity Monitor's graph shows)
linux cgroup v2 if limited, else /proc/meminfo MemAvailable
everything else inert, exactly as before
Two Linux details that matter, both covered by tests:
- MemAvailable, not MemFree. Linux fills RAM with reclaimable cache, so
MemFree reads ~1% on a perfectly healthy box. Using it would have wedged
spawns in block mode for no reason.
- cgroup v2 is consulted first so a containerised agent server is measured
against its own limit, not the host's RAM — otherwise a 2GB container on a
64GB host looks infinitely roomy right up until the OOM killer fires. And
inactive_file is subtracted from memory.current, because page cache is
reclaimable; without that a long-running container sits at 0% free forever.
PSI (/proc/pressure/memory) is deliberately not used: it is the sharper Linux
pressure signal, but it measures stall time rather than a fraction of memory, so
it cannot share TY_MEMORY_GUARD_MIN_FREE_PCT's meaning with the macOS path. One
threshold that means the same thing on every platform is worth more here.
The parsers are in a build-tag-free file so they are unit-testable on macOS
rather than shipped unverified; 11 new tests cover them. Verified building for
darwin, linux, freebsd and openbsd. (Windows still fails to build, on
executorlock's syscall.Flock — pre-existing and untouched by this change.)
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
govulncheck flags an infinite loop on invalid input in x/text v0.37.0 (GO-2026-5970), fixed in v0.39.0. It is an indirect dependency, but govulncheck finds a reachable symbol trace, so it reports as affecting this module. Pre-existing on main — the Vulnerability Check job started failing on 2026-07-22 and passed on 07-21, so the advisory landed in between. Folded in here rather than left for a separate PR so CI on this branch is clean. Only x/text moves in go.mod; the additional go.sum entries are module-graph checksums pulled in by x/text's own go.mod, not new build dependencies. Verified with the same govulncheck the workflow pins (v1.3.0): the x/text finding is gone and no non-stdlib findings remain. The stdlib findings that still show locally are an artifact of a local go1.25.5 toolchain — CI sets check-latest so it builds against the newest patched 1.25.x. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Nothing in ty bounded how many agent sessions could be live at once. Each session is a
claudeprocess plus its MCP servers, so a queue that fans out freely can walk a workstation into swap thrashing — at which point every session, dev server and test run on the box gets slower, including the ones already doing useful work.Measured on a 24GB M4 while diagnosing this: 36 live sessions across 393 processes, of which 35 session trees belonged to tasks that were already
doneorblocked— 4.4GB resident and a large share of what was filling 27GB of swap. Only 2 tasks were actuallyprocessing.What this is not
Deliberately not a concurrency cap. A fixed "max N sessions" is a hidden wall — the right number depends entirely on how much RAM the machine has and what else is running, and someone who legitimately wants dozens of concurrent agents shouldn't be told "no" by an arbitrary constant.
So this gates on the machine's actual memory pressure instead. With headroom, spawn as many as you like. The guard only has an opinion once the kernel says memory is genuinely short.
How
systemFreeMemoryPctis implemented per platform behind one stable contract — percent of memory still available, or "can't tell" — so the threshold means the same thing everywhere:kern.memorystatus_level— what Activity Monitor's memory-pressure graph is derived from/proc/meminfoMemAvailableOn macOS the default of 20 maps to Activity Monitor's zones (green 100-50, yellow 50-30, red 30-0), so it fires only well into the red — when the machine is already hurting, not merely busy.
Default mode is
warn: it logs loudly and never blocks. Behaviour is unchanged for everyone unless they opt in.TestGuardMemoryForSpawnDefaultNeverBlockspins that property specifically — it forces the threshold to 100% (so the machine is guaranteed "under pressure") and asserts the default mode still returns no error.In
blockmode the spawn returnsErrMemoryPressurecarrying the actual free percentage, the threshold, the task id, and the override — so if it ever does bite, it says why and how to get past it rather than silently stalling.Wired into both spawn choke points:
createTmuxWindow(daemon path) andEnsureTaskWindow(TUI/API path).Two Linux details that matter
Both are load-bearing, and both have tests:
MemAvailable, notMemFree. Linux fills RAM with reclaimable cache, soMemFreereads ~1% on a perfectly healthy box. Using it would wedge spawns in block mode for no reason. Pre-3.14 kernels fall back toMemFree + Buffers + Cached.inactive_file. A containerised agent server must be measured against its own limit, not the host's RAM — otherwise a 2GB container on a 64GB host looks infinitely roomy right up until the OOM killer fires. And becausememory.currentcounts page cache, which is reclaimable, a long-running container would otherwise sit at 0% free forever and stop spawning entirely.PSI (
/proc/pressure/memory) is deliberately not used. It's the sharper Linux pressure signal, but it measures stall time rather than a fraction of memory, so it can't shareTY_MEMORY_GUARD_MIN_FREE_PCT's meaning with the macOS path. One threshold that means the same thing on every platform seemed worth more than a slightly better Linux signal.Testing
gofmtandgo vetclean; fullinternal/executorsuite passes. Verified building for darwin, linux, freebsd and openbsd. (Windows still doesn't build, onexecutorlock'ssyscall.Flock— pre-existing, confirmed by building the branch without these files, and untouched here.)17 tests. The Linux parsers live in a build-tag-free file specifically so they're unit-testable on the macOS machines this gets developed on, rather than shipped unverified:
warn, neverblock), threshold parsing and out-of-range fallback,offbeing inert, the default-never-blocks property, block mode at 100% and 0% thresholds, raw signal in range.SwapCachednot leaking intoCached, garbage rejection.max) falling back, garbage rejection, over-limit clamping.Tests skip rather than fail where the live signal is unavailable, so they're safe on any CI.
Follow-up not included here
A
--strict-mcp-configchange was considered to cut per-session MCP fan-out (~21 servers merge into every task session today, only 1 of which istaskyou). It was rejected: strict mode's mechanism is "ignore everything except this file," which breaks ty's premise that a worktree agent has the same MCP servers its parent repo has. Worth revisiting only if Claude Code grows a way to disable user/plugin scope while keeping project scope.🤖 Generated with Claude Code