Machine Ceiling

The machine has a ceiling too

Section 11 of the method. Queue heavy commands per toolchain and run agent work at the lowest priority.

Section 6 caps changes in flight because of the merge queue. There is a second ceiling, and it is the machine the agents run on.

On 13 September 2026 I had thirteen agents running at once on a three core, 16 GB box. Every brief was reasonable on its own. Five of them were building the same .NET solution and starting Postgres containers for integration tests; the rest were installing npm packages and driving Playwright. Load average reached 44. The same box serves live sites. They held up, the static site still answered in about a tenth of a second, but that was down to what those sites are, not to any plan.

Nothing in this method said how many agents a machine can carry, because I had only ever measured tokens. Cost per change looked fine. The cost that was climbing was not on the bill.

Check before you fan out. nproc and uptime, before the first agent starts, not after the fans do.

Queue the heavy commands, not the agents. Reading code and writing a change is cheap. Compiling, installing packages, starting containers and running a browser is what saturates a machine. So let every agent run, and put only those commands in a queue: one lane per toolchain, one command per lane. A .NET build and an npm install can overlap; two builds cannot.

Run agent work at the lowest priority. nice -n 19 means a live service wins every contest for CPU. For processes already running, renice does the same without restarting anything.

Stop build servers from lingering. MSBuild keeps worker nodes alive after a build for the next one. With several agents that is a dozen idle processes holding memory. -nodeReuse:false makes them exit.

After lowering the priority of what was running and putting the builds in a queue, load roughly halved within a few minutes. Two agents finishing in that window helped, so I will not claim the whole drop for the fix.

The mistake worth naming is how I first applied it. I sent the rule to twelve running agents as a message. That is the instruction form section 4 warns against: it has to reach every agent, and the next one I start will not have it. heavy.sh in this repository is the script form.

bash heavy.sh dotnet dotnet build MySolution.sln -nodeReuse:false
bash heavy.sh node   npm ci

It takes a lane lock with flock, runs the command under nice -n 19, says so when it is waiting, and fails when handed no command instead of taking the lock and exiting 0. Name it once in the agent brief; the queue does the rest.

The first version of that script had a bug the same night. It took the lock with a plain flock, so every process the command started inherited the lock. dotnet build starts the Roslyn compiler server, which stays up for minutes after the build exits, and it kept the lock the whole time. Four agents waited about ten minutes on a lane where nothing was running. flock -o closes the lock before the command starts, so only the command itself holds it. The test that proved it: a job that leaves a 6 second child running made the next job wait 6.0 seconds before the change and 0.0 after, while two jobs in the same lane still took 4.0 seconds for two 2 second sleeps.

The general form is the one from section 10. A lock that is released when its holder exits is only as reliable as your knowledge of every process that can become its holder.

---

Previous: A check that passes having checked nothing | Home | Next: The adversarial critic

The plugin does this for tests with the Test Queue: agents enqueue, and one drainer runs one heavy job at a time through heavy.sh.