Test Queue

The test queue

Agents never wait for the test lane in a loop. They enqueue a job and end the turn. One drainer runs the jobs.

An agent that waits with sleep or pgrep burns a tool call, hangs at the tool timeout, and can match its own shell. So the queue lives in Brain. brain queue add returns at once. bin/test-phase is the drainer: it takes the oldest ready job, runs it, sorts the failures, and repeats until nothing is ready. This is the script form of the machine ceiling.

Adding a job

brain queue add <repo-dir> <test-project> [--filter F] [--tier fast|full] [--after ID]
                [--cycle C] [--by NAME] [--project P] [--platform apple|auto|none] [--args=EXTRA]

It prints the job id.

• --tier full is the default and runs the whole project. --tier fast needs a --filter.
• A filter can name several classes joined by |, for example --filter 'OrderTests|InvoiceTests'.
• --after ID chains a job behind another. It waits until that job is done. If that job failed or was cancelled, the new job is cancelled with the reason.
• --cycle groups jobs, runs and items. brain list --cycle C reads them back.
• --args=EXTRA adds dotnet arguments to this job's commands. Write it with the equals sign so a leading dash is not read as an option.
• --platform auto is the default. It marks the job apple when the test project targets -maccatalyst, -ios or -macos. none clears that mark.

If the same job is already queued (same folder, project, filter, tier, --after and arguments), brain prints that job's id and does not add a second. Agents that ask for the same run share one build.

Managing the queue

brain queue list [--cycle C] [--status S]
brain queue bump ID
brain queue move ID up|down
brain queue cancel ID
brain queue route ID --to remote|local|any

• list --status queued shows jobs in the order they will run.
• bump moves a queued job, and the queued jobs chained after it, to the front. The last one bumped runs first.
• move swaps a queued job with its neighbour. It never puts a job ahead of the job it waits on.
• cancel drops a queued job. Jobs waiting on it are cancelled too.
• route picks the lanes that may take a queued job: remote for remote lanes only, local for this machine only, any for whichever lane is free. See Remote Lanes.

Only a queued job can be bumped, moved, cancelled or routed.

How test-phase drains

bin/test-phase [PLAN] [--cycle C] [--lane L] [--stall S] [--run-dir D] [--project NAME] [--branch B]

One drainer runs at a time. It holds a lock file (drainer.lock in the run directory base). A second test-phase says "a drainer is running; it will take what is queued" and exits 0.

The drainer takes the oldest queued job whose --after job is done, bumped jobs first. It runs the job, records the result, and takes the next. When nothing is ready it looks once more, so a job enqueued just as the drain ended is not left for the next drainer. Then it prints a table and exits. On start it fails any job left running by a dead drainer, with the reason "drainer died".

Every command goes through heavy.sh in one lane (dotnet by default), so two sessions on one machine queue instead of building together.

Exit 0 means nothing real failed and no job was cancelled during the drain. Exit 1 means otherwise. A job whose failures were all flaky finishes done, so the jobs after it still run.

A plan file

test-phase PLAN enqueues the plan first. One project per line, # starts a comment:

<repo-dir> <test project path> [fast-filter] [args=EXTRA]

A line with a fast filter becomes two jobs: the fast slice, then the whole project.

Tiers

• fast runs one filter. Under MTP the fast tier passes *fragment* to --filter-class. Under VSTest it passes FullyQualifiedName~ terms.
• full runs the project.
• rerun is not a queue tier. After a job fails, each failing class runs alone. A test that passes alone (exit 0, or some tests passed and its name is absent) is flaky. A test that fails again is real. If the rerun was cut off, or failed without naming the test or passing anything, the test is unverified.

A filter piece that starts with - is refused, so it cannot be read as an option.

Runner detection

The runner is VSTest unless global.json (found by walking up from the repo) sets test.runner to Microsoft.Testing.Platform, which means MTP. TEST_PHASE_RUNNER=mtp or vstest overrides it.

• MTP: dotnet test --project P -- --filter-class ... --output Detailed
• VSTest: dotnet test P --logger console;verbosity=normal --filter ...

Stalls

--stall S (default 600, from TEST_PHASE_STALL; 0 turns it off) sets HEAVY_STALL for the command. If the output shows no new progress for S seconds, heavy.sh stops the command and exits 76. Progress is a changed counter line or a line starting passed or failed. The clock starts when the command starts, so keep S above the build time.

A stall is filed as a note item, not a failure. The message is that it is likely load, not code.

What lands in brain

Case	Item
A test fails and fails again alone	test-fail, ref is the test name
A test fails and passes alone	flaky, ref is the test name
A rerun that proves nothing	test-fail, body says unverified
Non-zero exit, no failing test named	test-fail, ref run-failed: <project>, body "exit N with no failing test named" and the log tail
Stall	note

Items carry the cycle, branch and the id of the run that showed them. Test output is masked for passwords, API keys, bearer tokens and JWTs before it is stored, and the body goes in by stdin, not argv.

Run records

Each command is a row in brain's runs table: tier, runner, duration, seconds waiting for the lane, load average at start and end, passed, failed, skipped, stalled, exit code, and host. test-phase writes them with brain run-add. brain stats reads them: per suite run counts, mean lane wait, mean duration, flaky rate, and the five slowest runs. Reruns are not counted as runs in the suite figures. brain stats --json gives the same as JSON.

The orchestrator pattern

1. Agents code and build only. Each adds its jobs with brain queue add and ends its turn.
2. The orchestrator starts one bin/test-phase as a background task.
3. It gets one notice when that process exits.
4. It reads the result with brain list --cycle C, sends the fixes out, and enqueues again.

Run brain queue list before adding. If an earlier job for the same worktree is still queued, cancel it and add one merged job.

Options and environment

Name	Meaning	Default
--cycle	Cycle label for a plan's jobs	the run directory timestamp
--lane	heavy.sh lane	dotnet
--stall	Seconds without progress before a stop	TEST_PHASE_STALL, else 600
--run-dir	Where logs go	<base>/<timestamp>
--project	Brain project name for a plan's jobs	the main repository's name
--branch	Brain branch	the repo's current branch
LEAN_TEST_PHASE_DIR	Base for run directories and the lock	~/.local/share/lean/test-phase
TEST_PHASE_RUNNER	mtp or vstest	detected
TEST_PHASE_DOTNET_ARGS	Extra dotnet arguments for every job	none
HEAVY_WAIT	Seconds to wait for a busy lane	3600 here (300 in heavy.sh)
TEST_PHASE_REST	Seconds a remote lane rests after a lane problem	600
TEST_PHASE_REMOTES, TEST_PHASE_REMOTES_FILE	See Remote Lanes	none

Logs are under the run directory, one file per command.