Remote Lanes

Remote lanes

The test queue can drain on SSH hosts as well as on your machine. Each host is one more lane.

Test Queue runs one heavy job at a time on the local machine. With a remote host there are two lanes, so two jobs run at once: one local, one remote. A lane takes one job at a time. Only the test command runs on the host. Parsing, reruns, the stall watchdog and Brain stay local. The full setup is in docs/remote-lane.md. This page is the short version.

The config

~/.config/lean/remotes.ini, or the file named by TEST_PHASE_REMOTES_FILE. No file means no remote lanes. A bad file prints a warning and the drainer runs the local lane only.

[remote dev]
host = dev
workdir = ~/lean-runs
cpus = 2
memory = 8g
min_free_gb = 15

[repo app]
sidecars = db=postgres:16
env_file = ~/.config/lean/app.env
ready = pg_isready -h 127.0.0.1 -U postgres
docker = no

• [remote NAME] needs host, an ssh config Host name. Other keys: workdir, cpus, memory, min_free_gb, cache_max_gb, image, extra_paths, exclude.
• [repo NAME] is named after the brain project of the job. Keys: sidecars, env_file, ready, docker.

Before a claim

Before each claim a remote lane sweeps its host for labelled containers, networks and job-* folders that belong to no running job. Then it checks that the host answers over ssh, has more than min_free_gb free in the workdir, and has docker running. If a check fails, the lane rests and says why. The local lane keeps going.

Running a job

The lane copies the job's worktree to the host: the files git ls-files lists (tracked and untracked, not ignored), plus extra_paths, minus anything matching exclude. The host does not need git. Then it runs the same test command in a docker container with the cpus and memory caps, and streams the output back into the local log. Everything a job creates carries the label lean.job=<id> and is removed when the job ends.

Secrets go only through env_file. It is sent as a mode 600 file and loaded into the test container, never put on a command line.

Sidecars in a [repo] section start on a private network for the job and are reachable by their alias. ready runs until it succeeds twice in a row before the tests start.

Testcontainers

docker = yes in the [repo] section gives the job its own Docker daemon (a docker:27-dind sidecar). Containers the tests start live inside that daemon, so they never touch containers already on the host, and they go when the job's sidecar goes.

Which jobs go where

A job whose test project targets an Apple platform is marked platform=apple and stays on the local lane. Use brain queue route ID --to remote|local|any to choose for a queued job. A job routed to remote with no remote lane up stays queued, and test-phase names it at the end of the drain.

When a lane fails

If ssh, the copy or docker fails before the test command starts, the job goes back to queued. No failure is filed. The lane rests for TEST_PHASE_REST seconds (default 600), and the local lane or another host takes the job. The rest ends early once no lane is working. If the connection drops during a class rerun, the unfinished classes are marked unverified.

TEST_PHASE_REMOTES=off turns every remote lane off.

Setting up a host

1. ssh access with a key, no password prompt, and the host key already in known_hosts.
2. docker usable by that user, and rsync.
3. Free disk above min_free_gb.
4. A [remote NAME] section in remotes.ini.

Run one test-phase per host list. The sweep assumes every running job it sees in brain is its own, so two drainers sharing a host would remove each other's work.