pi-dispatch v4.0.0

pi-dispatch 4.0.0 gives each project's jobs their own size, and keeps the jobs running on a machine inside a budget of its memory and CPUs, with room left for the machine itself.

Until now every job container got the same 4 GB and 2 CPUs, swap on top, and nothing checked that the running jobs together fit the machine. A heavy project could not get more, a light one could not get less, and a few busy jobs could slow every other job and the machine itself to a crawl. 4.0.0 measures what each run uses, lets you give each project a size, admits a job only when its size fits what is left on the machine, and, where the container runtime allows it, keeps one CPU core free for the machine across all jobs. It also suggests sizes from the measured runs.

This is a major version because defaults change what runs:

• a job no longer gets swap beyond its memory;
• with the automatic budget, a small machine runs fewer jobs at once than PI_CONCURRENCY alone allowed;
• one job is no longer held to 2 CPUs.

It also closes a hole, open since 3.0.0, that let a job read the key that signs its result. Read Before you upgrade below before you restart anything.

What is new, and why

Every run records what it used

Why: a size is only as good as the measurement behind it.

What:

• Each run record carries resources: peak memory and swap, out of memory kills, memory pressure, CPU time and throttling, and the peak number of processes. The job reads these from its own container and signs them into its result line.
• Records also carry size (the memory and CPUs the job got, and where that came from). docs/insights.md explains every field.

A job killed for running out of memory is no longer retried and paid twice

Why: an out of memory kill looked like any other crash (exit 137), so it was retried at the same size, failed again and was charged twice.

What:

• A small supervisor in the job image reports a kill the job itself cannot report. The kill is recorded as oom-killed, not retried, and the forge comment says the job ran out of memory. Your failure hook receives it too.
• It counts only when it is confirmed: the signed line, exit 137, a kill counted by the kernel, and a peak that reached at least 90% of the job's memory. A machine that ran short as a whole, below the job's own limit, is still retried as before.

A size per project

Why: projects differ: one may need 20 GB for its builds while another runs fine in 2 GB.

What:

• A project:<id> row in the scoped limits file can set memory (for example "12g", at least 512m) and cpus (for example 0.5 or 4, at least 0.25). Other jobs use PI_JOB_MEMORY and PI_JOB_CPUS, which default to 4g and 2.
• A size needs version 3 of the scoped limits file. A version 3 row refuses a key it does not know, so a typo such as Memory is refused instead of ignored.
• dispatch_limit_add and dispatch_limit_edit take memory, cpus, hostShare and minJobs, behind your confirm.
• Under contention a larger CPU size gets more CPU. A job alone may use idle cores up to the machine's CPU budget.
• Shared memory (/dev/shm) is half the memory, at most 1 GB. The process limit stays at 512.

A budget per machine

Why: a job count cannot tell a 20 GB job from a 2 GB one, so a count alone either wastes a big machine or overloads a small one.

What:

• A job starts only when its memory, its CPUs and one job slot fit what is left. PI_CONCURRENCY is now the budget's job count.
• PI_HOST_MEMORY_BUDGET and PI_HOST_CPU_BUDGET default to auto: the memory and CPUs the container runtime reports (on Docker Desktop, the VM's), minus a reserve for the machine (10% of the memory, between 1 GB and 4 GB, and one CPU on a machine with 4 or more), never less than one job of the default size. Set a value or off in .env; the worker reads them only when it starts.
• A job's cpus count against the CPU budget, although the runtime uses them as a weight. So a 4 CPU machine (budget 3) runs one default job of 2 CPUs at a time.
• No starvation. The oldest waiting job keeps its place, so a stream of small jobs cannot keep a big one waiting forever.
• Fairness. minJobs keeps room on each machine for a project's oldest waiting job until that many of its jobs run there (it needs memory or cpus on the same row). hostShare caps the memory and CPU a project's running jobs may hold, as a percentage of each machine's budget.
• A size that can never fit is refused before anything is spent, naming both sizes in the record (job-size-exceeds-host, job-size-exceeds-share). On a machine in a fleet (PI_WORKER_NAME set), a job on the shared queue waits for a machine it fits on instead.
• Restart safe. A restarted worker counts the job containers still running before it starts new ones. If it cannot list them, it starts no job until it can, and doctor says so.

One CPU core kept free across all jobs

Why: each job's CPU limit applies to that job alone, so busy jobs together could take every core and leave the machine, the egress proxy and Valkey only a share.

What:

• Every job container runs in one group, pidispatch.slice, whose CPU limit is the machine's CPU budget. Measured on Docker Desktop, Docker on Ubuntu, and Podman 4.9 and 5.8: three busy jobs together stayed at the budget, and a busy program outside the jobs kept its core.
• The worker sets that limit itself on rootless Podman (through your account's systemd user manager, a setting that survives a reboot) and on Docker Desktop and Docker's cgroupfs driver (through a short helper container of the job image, with every capability dropped and no network).
• Docker on Linux with the systemd driver, and rootful Podman, need root once. pi-dispatch doctor prints the exact command.
• On a cgroup v1 host no limit is kept. Where rootless Podman uses the cgroupfs cgroup manager, jobs run outside the group, and each job's CPU weight is capped so it cannot outweigh the egress proxy and Valkey.

Size suggestions from the measured runs

Why: measured runs are better evidence than a guess, but they come from inside the job, so they advise and never decide.

What:

• doctor, the panel's PROJECTS view, the dispatch_limit_edit preview and the insights page show a suggested size per project, with the exact call to apply it.
• Memory is raised only after a confirmed out of memory kill. It is lowered only after at least 10 runs, and never below 1.25 times the largest peak of the last 30 days. CPU is only ever lowered.
• A suggestion never goes beyond what the machine, or the project's share of it, offers. Nothing is applied automatically.

The key that signs a job's result could be read by the job itself

Why: the worker trusts a job's result (tokens, cost, outcome) because the runner signs it with a key the job cannot see. Review found that code inside the job could send the runner a signal that opens Node's debugger on the container's loopback, take a memory snapshot and read the key. With it, a job could have signed a lower cost. This was open since the signing was added in 3.0.0.

What:

• Both processes that hold the key now start with the debugger closed to other processes.
• The image check (verify-image.sh) tries to open the debugger in every process that holds the key, and fails the image if any opens.
• Two routes remain, both older than 4.0.0, and the spec names them: code loaded into the runner itself (the serviced repository's .pi/extensions, taken from its default branch), and a NODE_OPTIONS you set yourself (a trigger secret of that name, or PI_FORWARD_ENV).

Jobs run on pi 1.0.4

What:

• pi's exclusion list now also takes * patterns. The excludeTools field in triggers still takes only tool names, and refuses a pattern like a misspelling.
• Two openrouter models left pi's catalog (qwen/qwen3.8-27b:free, stealth/space-bunny-alpha), and 14 models changed price.

A guide to sizing jobs

docs/sizing.md walks through all of this in order, with a worked example: a 64 GB machine shared by a heavy, a medium and a light project.

Before you upgrade

1. Move the job image with the worker. Pull or build the 4.0.0 job image. Without it, runs record no measurements, out of memory kills are retried as before, the debugger hole stays open, and jobs keep running pi 1.0.3. If you build your own image from this one, keep this repository's ENTRYPOINT: it now starts a small supervisor that starts the runner. An entrypoint that starts the runner directly still runs, but its out of memory kills are retried, and verify-image.sh now fails it while it declares exitAuth. The supervisor uses about 8 MB of the job's own memory.
2. Upgrade and restart every worker, and the panel, before you write a size. An older worker refuses a version 3 scoped limits file when it starts. One that is already running keeps its last good file: the size, and every later edit to that file, do not apply on it. Once the file is version 3, doctor names every registered worker that cannot read it. Update the panel too (pi install npm:@edgehero/pi-dispatch-admin): a 3.1.0 panel shows a version 3 file as invalid and cannot edit it.
3. A scoped limits file at version 1 or 2 that carries memory, cpus, hostShare or minJobs on any row (keys older versions ignored) now stops the worker at boot. Remove them, or move the file to version 3 with the size on a project: row.
4. No swap beyond memory. A job that used to get by by swapping is now stopped at its memory and recorded oom-killed. If that happens, give its project a larger memory (or raise PI_JOB_MEMORY for jobs outside a project); doctor suggests a size per project. Where Docker reports SwapLimit false the swap limit is not enforced, and doctor warns.
5. Fewer jobs at once on a small machine. With auto, a Docker Desktop VM with 8 GB runs one default 4 GB job at a time, and so does any machine with 4 CPUs or fewer (a 4 CPU machine has a CPU budget of 3, and a default job counts 2). doctor says whether the budget or PI_CONCURRENCY binds. To run more, give the VM more memory, lower PI_JOB_MEMORY or PI_JOB_CPUS (or a project's size), or let only PI_CONCURRENCY bound the count again with PI_HOST_MEMORY_BUDGET=off and PI_HOST_CPU_BUDGET=off in .env. The worker reads these at start, so restart it.
6. One job is no longer held to 2 CPUs. A job's cpus is now its share under contention, and on an idle machine one job may use every core of the CPU budget. To keep jobs off other work on the machine, set PI_HOST_CPU_BUDGET to a value.
7. Docker on Linux with the systemd driver, or rootful Podman: run one command. Without it no CPU core is kept free across jobs. Run pi-dispatch doctor and run the sudo systemctl set-property pidispatch.slice CPUQuota=... line it prints, once per machine, on the machine the containers run on. It survives reboots. Run it again if the CPU budget changes. On rootful Podman and on a Docker daemon on another machine, doctor cannot read the limit back, so its warning stays after you ran it.
8. Thread jobs start fresh once. A run.resume job never resumes a session another pi version wrote, so the first resume after the upgrade starts a new session and is paid in full.
9. The receiver package moves to 4.0.0, to depend on the 4.0.0 worker package. Its own code did not change.

Sizing guide: docs/sizing.md. Issue: #596.

Full changelog: v3.1.0...v4.0.0