Field Notes

Field notes from parallel agents

Sections 14a to 14h of the method. Leftover processes, shared scratchpads, wait loops, stacked pull requests, deploys, rehearsals, local CI, steps for a person.

14a. Sweep for what parallel agents leave behind, because none of it announces itself

Four to six agents running in parallel for one day left, all of it silently:

• twelve shells polling forever on a pattern that matched their own command line, keeping each other alive;
• three more waiting on work that had been overtaken, making about eight hundred pointless API calls;
• four web servers orphaned nine days earlier by sessions that had ended, listening on every interface;
• two agents on the same fixed test port, each driving the other's code;
• two agents publishing each other's pull request bodies, which left an issue open whose work had merged;
• about sixty five gigabytes of worktrees on branches that had all merged.

Not one of these showed up as a failure. That is the whole problem. A stuck wait and a working one look identical in a task list. An orphaned server answers requests normally. A worktree on a merged branch looks exactly like one on live work. A test run that reuses somebody else's server goes green.

So this is not a rule to remember, because remembering is what failed. It is agent-hygiene.sh, run between batches. It reports and never deletes, because deciding what is live is the part a script cannot do.

./agent-hygiene.sh                    # the current repository
./agent-hygiene.sh ~/dev/a ~/dev/b    # several

Two things learned writing it. Container workloads are excluded by parent process, because a poller inside a container is doing its job and a check that reports it every run teaches people to skip the output. And merged state has to come from the pull request, never from a commit count, because a squash merge leaves the original commits looking unmerged and a count will tell you nine branches have unlanded work when they have none.

14b. Give every agent its own scratchpad directory

Agents running in one session share a scratchpad. Two of them wrote a pull request body to scratchpad/pr-body.md, and the second write published the first agent's body onto the wrong pull request. It happened twice in one day.

The first time cost a real thing. The overwritten body carried the line that closes an issue, so the issue stayed open although its work had merged, and the merged pull request kept a permanent record of a change it did not contain. Nobody noticed for hours, and only then because the issue count looked wrong.

The second time an agent caught it on a final check before handing back, and said the thing worth repeating: this one was visible because a pull request body gets published. A plan file or a notes file clobbered the same way is just quietly wrong, and nothing ever shows you.

So: every agent writes under its own subdirectory, named for the agent or the branch, and never at the root of the shared scratchpad. Bare names are the trap, because every agent independently invents the same three: pr-body.md, notes.md, plan.md.

One line in the brief does it:

Write every file you create under scratchpad/<your-branch-name>/. Never write to the root of the scratchpad, and never use a bare name like pr-body.md, because another agent is using the same directory and will pick the same name.

This is the same class as the fixed test port in section 14: shared mutable state with a predictable name, where the failure is silent and looks like success. Worth looking for wherever agents run in parallel, because they do not collide randomly. They collide on the obvious name.

The same goes for a file every change appends to. Parallel agents adding entries to one changelog left conflict markers and duplicate entries. One fragment file per change, assembled at release, gives them nothing to collide on.

An agent in its own worktree has one more rule: it runs git as plain single commands from the worktree root, with no cd x && in front and no pipe behind. The harness refuses a command it cannot verify, and a compound one is exactly that.

14c. Every wait loop needs a deadline, not just a condition

Agents wait on things: a test run, a CI job, a pull request's checks. They write the obvious loop, which polls until a condition holds, and the loop is correct. It is also unbounded, and the thing it waits for can stop being reachable while it waits.

This happened twice in one day, with different causes and the same shape.

The first time, twelve shells polled with pgrep -f 'dotnet test ...'. That pattern matched each shell's own command line, so every one of them found itself, concluded the suite was still running, and slept again. They kept each other alive too. The oldest ran for nearly three hours.

The second time, three shells waited on work that had been overtaken. One waited for a commit to appear in a CI run, but that commit never landed because the pull request merged while it was in flight. One waited for a merged pull request's head to change, which it never will. One waited for a file to disappear that is still there. Between them they made roughly eight hundred pointless API calls over three and a half hours.

None of this shows up as a failure. The task list just shows something waiting patiently, which is indistinguishable from something working.

Three rules, in order of how much they buy:

1. Bound every loop. A deadline turns a stuck wait into a report. for i in $(seq 1 60); do ... done and then say plainly that it timed out, rather than until ...; do sleep; done.
2. Never match on a command string that your own command line contains. Watch a file the run writes, check a process id you captured when you started it, or exclude $$.
3. Re-check the premise, not only the condition. A loop waiting for a pull request should stop when that pull request merges, whatever its head commit says. The condition and the reason for waiting are different things, and only the first one is usually written down.

I wrote here that the harness re-invokes an agent when tracked work finishes, so most of these loops should not exist. That is true for the session I talk to. It is wrong for a subagent: a subagent is never told that background work finished, so it waits on something long done and stalls. So a subagent runs its commands in the foreground, and every piece of a gate is sized to finish under the tool timeout of 600 seconds, counting any time spent waiting for a lock. When a loop genuinely must exist, it gets a deadline.

14d. Landing a stack of pull requests is its own job

Seven pull requests, each built on the one before, all reviewed and green. Squash merging the bottom one broke the next in four different ways over one morning.

• A squash merge leaves the next branch holding commits the default branch no longer has, so it shows as conflicting even though nothing conflicts. Merging the default branch in is safe only when the squashed tree is a state the next branch already passed through; then keeping the branch's own tree loses nothing and keeps exactly what was reviewed. Check it, do not assume it: compare the squash commit's tree with the trees in the branch's history.
• Deleting a base branch can close the pull request on top of it rather than retarget it. Retarget the next pull request to the default branch first, then delete. A closed one can be reopened only after its base branch is pushed back.
• Required checks do not rerun when only the base changes. One pull request sat blocked with every check it had run passing, because the one it lacked had never been triggered for its new base. Closing and reopening it ran it.
• grep -q under set -o pipefail can report a match as a failure when the command feeding it has more to write: grep exits on the first match, the writer dies of a broken pipe, and the pipeline fails. The check meant to prove the stack was safe stopped it, twice, for no reason. Send grep's output to /dev/null instead of using -q in a pipefail script.

Write the landing as a script with those checks and stop at the first surprise. Landing a stack by hand makes the same four mistakes, one at a time.

One more thing keeps pull requests from landing, and nobody owns it. A fix lands on the default branch, and every open pull request that was red for the same reason stays red on its old base until somebody notices. templates/refresh-stale-prs.yml is a workflow for that: when the default branch moves, it updates the branch of every open pull request that is behind it and failing. It never retries a job, because a retry hides a flake, while a newer base rules out one cause and leaves a real failure visible. It never merges, skips forks and drafts, and leaves alone any pull request labelled no-self-heal. Each pull request is refreshed once per failure: if its head is already a merge the workflow made and it is still red, the failure is its own, and it is left for a person.

14e. Test what you published, not what you built

A release moved the types every module is compiled against into a new package. Namespaces were unchanged, so every module in the repository still built, every test passed, and the changelog said no module needed an edit. The modules already published, compiled against the old release, could not load, and a host that referenced any of them did not start at all. Nothing in the repository could see it, because the repository only ever built from source.

It was found by installing the published packages into a fresh host, which is one CI job. The fix was type forwarders, which the project's own public API rule required; the missed step was never testing the thing users install.

If a release changes where public types live, a job that installs the previous release's published packages against the new build is the only check that sees it.

The same holds after a deploy: assert content and identity, not status codes or ratios.

• A page baked empty still answers 200. A fresh image prerendered while its data source was unreachable, so its pages carried an error notice instead of content until the next revalidation, and every one answered 200. A status check called it a good release. The check now looks for text the page must contain.
• A ratio hides what it averages. A pixel comparison on a tall page reported 45 wrong dates as 0.005 percent of the pixels, under its cap, and passed. Read the diff image, not the number.
• A version string is not an identity. Today's 0.1.0 and yesterday's 0.1.0 are the same string and can be different builds. Bake the commit sha in at build time and compare that against the commit you meant to deploy.

14f. A rehearsal proves only what it copies

A production cutover was rehearsed five times on a copy of the production database, with the rollback proven three times. Two things still went wrong, and neither was in the copy.

The production directory belonged to another user, so the script could change the files in it but not create one beside them. The rehearsal ran in directories the script owned. A reviewer found it by reading the live directory's permissions, before the real run, and the next rehearsal made its copy belong to another user on purpose.

The script exported an address for its own calls to the API, the one a process on the host uses. A compose run later in the same shell inherited it, and compose fills a variable from the shell before the file beside it, so the renderer started pointed at its own container and every page answered 404 for a few minutes. The rehearsal never used the address the production file held, so it never met the collision. The fix is a wrapper that strips the script's own variables before calling compose, with a test that fails if one gets through.

Copy the environment as well as the data: ownership, the variables in the shell, the network the containers share. A rehearsal that differs in any of them is a rehearsal of something else.

14g. When the CI minutes run out, the workflow still decides

A private repository ran out of free CI minutes in a month: 1,903 of 2,000, nearly all from one site's checks on every push. Turning its CI off was the right call, and it left the question of what counts as green.

The answer was to keep the workflow file as the definition and run its jobs locally: each job in a clean checkout of the commit, every run step in order with the shell the hosted runner uses, stopping at the first failure. A pull request is green when that run passes at its head, and the output goes in the pull request. Turning hosted CI back on changes nothing, because the definition never moved.

The first version was a script in the repository. It got three things wrong that a hosted runner gets right: pipefail on steps that do not ask for it, a missing runner temp directory, and not rewriting a port inside a step's environment. As a separate tool with its own tests it got all three right. A local runner needs its own tests for the same reason a check does: one that passes having checked nothing is worse than none.

14h. Plan the steps that need a person

Some steps are not an agent's to take: a merge with no standing permission, anything in production, a write to an outside service, a force push or a deleted branch. Found at the end, each one is a stall: an agent waiting, or a hand-back that says a person is needed and gives no command.

So they get planned at the start, like the tickets. Before a batch begins, list the steps that need a person. When the person gives a standing permission, such as "merge each pull request of this release once CI is green and the review passes", write it down with its scope: which pull requests, which repositories, until when, and what it does not cover. Production stayed outside it even then.

A run ends with one block of exact commands for the person, in order, ready to paste, each with what it proves when it succeeds. Not a list of things to do. Turning a summary back into commands is the step that goes wrong.

---

Previous: Cheap drafting works, and it stops short | Home | Next: What a reader should observe