Repository navigation
Releases: ktanaka101/loadmaster
Release list
v0.17.0
loadmaster says to an agent only what it can act on, and does itself what only it can: a waiting run no longer fills the logs agents read later, a check that just failed under load outside the queue waits for the machine to have room before running again, a guard test's held refusal is said only when it may be why a check failed, and status shows this checkout's jobs and what they wait behind. Measured with fresh Claude Code and Codex sessions on held-out scenarios against 0.16.1; see eval/agents.
Changed
- A wait read later says one line. Off a terminal, where an agent's tool hands the output over once the command has ended, a waiting run says what it waits for and behind whom when it starts waiting, then only
loadmaster: #42testhas waited 4m00s;loadmaster statussays what forevery 4 minutes, to keep the command alive. On a terminal, as before. In the held-out logs, what agents read fell to 28%. - A check that failed under outside load waits it out. If the last run of a task from this checkout failed or was killed while the CPUs were 95% busy or more, ran 3 s or more and ended in the last 30 minutes, the next run waits before getting in line until the CPU outside the queue leaves it its weight: at most 10 minutes, then it gets in line anyway; within
--max-wait, after which it exits with 75 and keeps no place. A run that may resume a kept place does not wait. The note after such a failure now ends with aloadmaster: next:line saying so, and a failure in under 3 s gets no note. Agents had run such checks again at once into the same load or handed the wait back to their user: on the held-out scenario 1 of 6 runs got the check to pass, now 6 of 6. loadmaster heldinside a job leaves the reason to the job's run. In the process group of a running job, a refusal is the asking script's:heldprints nothing then, keeps the reason in the state directory, and the job's run says it (loadmaster: note: inside it, ...) at its end only if the job failed. Outside any job it says why and what to run, as before. A test that checks a guard refuses no longer prints the refusal in every passing run, which agents had reported to their user as a warning.statusin a checkout shows its jobs and what they wait behind. Run from a checkout, the text lists that checkout's jobs and, for each that waits, the jobs ahead of it that have not started and the jobs its wait names, with the jobs using the cores; the rest is one line. A job of this checkout isfrom this checkout. Agents found their own job and why it waited in one call, reading 10 to 41% of the status output of 0.16.1.- The guide says the new wait lines, what to do after the note, and that
--jsonis for programs: Codex, told it was for tools, read the JSON, several times the text.
Added
loadmaster status --all: every job, asstatusshowed before (also what it shows outside any checkout).eval/agents: the harness that measured these changes, with its scenarios, thresholds and results.
Breaking changes
status(text) lists only this checkout's jobs and those in their way when run from a checkout; use--allfor every job.status --jsonis unchanged and lists every job.- Off a terminal, a waiting
runno longer says when what it waits for changes, nor once a minute. loadmaster heldprints nothing when it refuses inside a running job's process group; its exit code is unchanged.
Install
mise use -g github:ktanaka101/loadmaster@0.17.0Or download the archive for your platform below: macOS (Apple Silicon and Intel) and Linux (x86_64 and arm64, statically linked).
v0.16.1
loadmaster doctor no longer asks a repository whose own commands go through loadmaster to add an instruction line.
Fixed
- doctor asks for no change that is not needed. A repository whose
package.json,Makefile,justfile,Taskfile,mise.tomlordeno.jsonruns its checks withloadmaster run, or guards them withloadmaster held, puts an agent that uses those commands in the queue without telling it. 0.16.0 still told such a repository to add a line toAGENTS.md, which an agent following doctor might do; now the instructions check isokthere, and says why.
Install
mise use -g github:ktanaka101/loadmaster@0.16.1Or download the archive for your platform below: macOS (Apple Silicon and Intel) and Linux (x86_64 and arm64, statically linked).
v0.16.0
loadmaster is made for AI agents as its users. Every line says what happened and, where there is something to do, exactly what to run next; the binary tells an agent how to use it here (loadmaster guide), checks that its tool lets loadmaster work (loadmaster doctor), and replays the machine's history with other weights before it changes them (loadmaster tune). Nothing it could do before was taken away.
Added
loadmaster guide: how an agent runs checks here, from the binary itself, so it matches the version installed: the rules (only through loadmaster, one command per call, waiting is normal, what each way a run ends means and what to do, how to stop one's own checks), the machine's capacity and mode, and this repository's tasks, one line each, with their weight, the cores they may start with, resources, the exit codes that pass, their usual run time and theirdescription. It says the rules where there is no config, or a broken one. An instruction file needs one line:Run checks only through loadmaster: before running any, runloadmaster guideand follow it.loadmaster doctor: whether loadmaster works for agents here, and what to change where it does not: this binary against the one onPATH, the configs,LOADMASTER_MAX_WAIT, whether this process may write to the state directory (run by an agent's tool, the sure test of its sandbox), the queue file, Claude Code's sandbox and timeout settings, Codex's sandbox, and whetherAGENTS.mdorCLAUDE.mdtells agents of loadmaster. A finding isproblem(exit code 1),note,unknownorok, as sure as what it rests on, with the file and the snippet to change. It changes no settings itself.--json.loadmaster tune: what other weights or min_weights of this repository's tasks would have done to this machine's checks, replayed through the admission rule itself: the current settings, measured weights (the 90th percentile of the cores each task's passed runs kept busy) and your own--set TASK.weight=N/TASK.min_weight=N. It compares waits, times from getting in line to the end, and the share of the time over capacity, prints the change to try, and states every assumption of its model.--json.- A task's
description, one line, for the agent choosing a task (the guide, and the error for an unknown task). report --json: the report as data, from the same rows as its table.status --json:same_checkoutandusual_runfor each job, andrecent: the checks of this checkout that ended in the last 30 minutes.
Changed
- A run names its job and says how it ended:
loadmaster: #42teststarted, and last,loadmaster: #42testfailed in 2m03s (exit 1; waited 12s): a task's own exit code of 75 or 125 is no longer taken for loadmaster's. One that gives up says what to run, word for word:loadmaster: next: runloadmaster run test -- a.tsagain from /repo within 120s: it resumes its place in line. Errors with one remedy end with aloadmaster: next:line (an unknown task gets the closest name). - A waiting run says less, but never nothing for long: when it starts waiting, when what it waits for or who holds it changes (at most every 15 s), and at least every 60 s (some tools end a command silent for 5 minutes), each time naming the running jobs it waits behind, how long they have run and how long they usually run. Before, a line every 10 s.
statusspeaks to agents: one line per job with what it needs (cores or weight, for how long and how long it usually runs, where from,(same checkout), and why it waits or how long a parked place is kept), no table padding, PIDs or long paths; then what ended lately from this checkout, for a run whose output was lost.- Usual run times and
tunecount a task by its repository, which all its worktrees share, and by its setting and arguments. The history records each check's checkout, workload and resource limits. LOADMASTER_QUIET=1leaves out the lines that only inform, so a task's stderr passes through as it is (for scripts: a tool that ends silent commands would end a quiet wait).
Breaking changes
- The words of
run,status(text) and errors changed.status --jsonkeeps format 2 and only gained fields. run's stderr now has two lines of its own around the task's output; setLOADMASTER_QUIET=1for none.
Install
mise use -g github:ktanaka101/loadmaster@0.16.0Or download the archive for your platform below: macOS (Apple Silicon and Intel) and Linux (x86_64 and arm64, statically linked).
v0.15.2
Signals reach a running task without depending on a look at its process, and a run counts as stopped only if its stop reached the task's own command while it ran.
Fixed
- A signal for a running task is never dropped by a misread. Before each signal, loadmaster checked the task's process group by its leader's start time, so that a group ID the OS reused after the task ended is never hit. While the task's command is a child not yet waited for, its PID (the group's ID) cannot be handed out again, so the group is the task's for certain: the signal now goes without that check, and the OS says whether anyone got it. Once the command has been waited for, the start time is checked as before. A start time misread while the task ran would have dropped the signal without a word; this was closed in case it was behind a Ctrl-C test that timed out once on a macOS Intel runner (never reproduced).
- A failure is no longer recorded as a stop when the stop came too late. The history's
stopped(asked to stop before it exited), whichreportcounts as killed rather than failed, took any stop sent before the command's exit was seen. A command that failed on its own just before a stop, leaving processes in its group, or that had moved itself to another group, was recorded as stopped when the stop reached what it left. Now loadmaster asks the OS, as the command's parent and without waiting for it, whether it still runs in its group right before the signal, and only a stop that reached it counts. That also holds where a look by PID cannot see the command (hidepidon Linux, a command that is not dumpable).
Tests
- The stop tests wait for at least the signals they expect, and on a timeout say what the task logged, whether it and loadmaster still run, and what loadmaster said.
Install
mise use -g github:ktanaka101/loadmaster@0.15.2Or download the archive for your platform below: macOS (Apple Silicon and Intel) and Linux (x86_64 and arm64, statically linked).
v0.15.1
Looking at a queue of 0.12 to 0.14 with 0.15 no longer stops the older agents: only a run of 0.15 starts moving it to the new protocol.
Fixed
- Only getting in line moves the queue. 0.15.0 closed a queue of 0.12 to 0.14 to new jobs as soon as any of its commands opened it, so
mise exec github:ktanaka101/loadmaster@0.15.0 -- loadmaster status, run to look before upgrading, made every run of the older agents fail. Now the firstloadmaster runof 0.15 to get in line starts the move, as it is what needs the new protocol: it moves an empty queue at once, and closes one with jobs in it until they are done, as before.status,held,cancel,cleanand nested runs work with the queue as it is: they reap, show and cancel the jobs of 0.12 to 0.14, which those admit by their own rule, andstatussays what the first run of 0.15 would do. status --json'sprotocolis the queue's: 1 for a queue of 0.12 to 0.14 until a run of 0.15 has moved it, even while it is empty.
Queues that 0.15.0 already moved or closed are not affected.
Install
mise use -g github:ktanaka101/loadmaster@0.15.1Or download the archive for your platform below: macOS (Apple Silicon and Intel) and Linux (x86_64 and arm64, statically linked).
v0.15.0
A task that sizes itself by LOADMASTER_WEIGHT can now start with the cores that are left instead of waiting for its whole weight: set min_weight. Every job that may start now starts at once, in line order. The queue moves to a new admission protocol for this, once the jobs of 0.12 to 0.14 in it are done: upgrade every agent on a machine at once.
Added
min_weight, per task: the fewest cores it may start with, from 1 to its weight. The job then starts with the most cores between that and its weight that fit beside what runs and waits ahead of it, andLOADMASTER_WEIGHTsays how many. Set it only for a task that sizes its workers byLOADMASTER_WEIGHT(vitest --maxWorkers=$LOADMASTER_WEIGHT), which runs longer with fewer: in a simulation of a day of agents' checks on 8 cores, it cut the mean time from getting in line to the end by 30 to 45%. A task without it always starts with its weight. A task that holds a scarce resource (a browser, a database) holds it longer with fewer cores, which may keep others waiting longer than it gains. Like a weight, amin_weightabove the machine's capacity counts as all of it, so a repository's settings work on a small CI runner too.- Jobs behind one that may start with fewer cores leave room for it to start with its
min_weight, and still count it at its weight. No job is delayed, or started with fewer cores, by the jobs that got in line after it, and the jobs that start never take the machine past a limit. status,status --json(protocol,moving, each job'smin_weight,wait.cores),check,check --scenario, the history andreport(a row of its own for eachmin_weight, shown as4 (from 1)) show the least each task may start with.
Changed
- Every job that may start now starts at once. The first waiting
loadmasterthat finds jobs that may start starts all of them, in line order, in one transaction; each one'sloadmasterfinds its job started on its next poll. Before, each job started on its own poll. - A run that gives up and is started meanwhile by another waiting
loadmasterruns instead of parking. - A job's time in line (the history's
queued_at,enqueued_at_msinstatus --json) counts from when its run asked for a place, so a wait while the queue moves counts as waiting.
Breaking changes
- The queue moves to protocol 2. The first 0.15 to open a queue of 0.12 to 0.14 closes it to new jobs: runs of 0.12 to 0.14 then fail and say to upgrade, and runs of 0.15 wait without a place in line, as long as
--max-waitallows (exit code 75, no place kept). The jobs already in it finish by their own loadmaster's rule; running jobs are never stopped. Once it is empty, the first 0.15 moves it, and 0.12 to 0.14 cannot use it any more. The queue file staysqueue-v4.db. - CPU busy outside the queue lowers what a job with
min_weightstarts with, never below itsmin_weight. A task withoutmin_weightnow always starts with its weight, never with fewer: give it amin_weightto have it start with fewer while the CPU is busy. - A nested
loadmaster runmust fit in the cores its holder got, not the holder's weight: a holder that started with 2 of 8 no longer runs an inner task of weight 4. An inner task withmin_weightruns if that fits, with the holder'sLOADMASTER_WEIGHT.
Install
mise use -g github:ktanaka101/loadmaster@0.15.0Or download the archive for your platform below: macOS (Apple Silicon and Intel) and Linux (x86_64 and arm64, statically linked).
v0.14.0
The queue file stays the same from now on, so upgrades no longer split the machine into two queues; status --json says when each job entered its state; check --scenario replays a captured queue by the admission rule; report counts runs that used more cores than they got.
Changed
- One queue file, for good. The queue stays
queue-v4.db. The database now records its layout and the admission protocol its writers keep, and every write of 0.14 and later checks both first: a loadmaster that does not know what a later one moved the queue to stops with an error instead of starting work, never with a new file of its own. Versions that keep the same protocol share the queue and count each other's jobs: 0.12, 0.13 and 0.14 do, so agents can move to 0.14 one at a time. A later change of protocol will close the queue to new jobs and switch only once it is empty, which stops 0.12 and 0.13 too. Only the files of 0.11 and before (queue-v3.dband earlier) stay separate;statusandrunnow list their running, waiting and parked jobs, and a file that cannot be read as one that may hold jobs. - When a job entered its state. The first 0.14 to open the queue brings its layout up to date in place, while 0.12 and 0.13 go on using it: each job records when it got in line or resumed its place, parked, started, or was handed over to its task's process group, kept by the database so that 0.12 and 0.13 keep it too.
status's FOR counts from it; where that time was not seen (a job left orphaned by a loadmaster that died, one waiting from before), it shows-. report: the new OVER column counts the passed runs that kept more than 1.2 times the cores they got busy, each against its own grant: a task that does not size itself byLOADMASTER_WEIGHTshows up when it got fewer cores while CPU was busy outside the queue, even within its weight.--args TEXTkeeps only the checks with an extra argument that contains TEXT, to look at runs that tried a flag.
Added
loadmaster check --scenario FILEjudges the queue a file describes, written asstatus --jsonwrites it (-for stdin), by the same admission rule, with the CPU busy outside the queue the file gives: capture a real queue, change what you want to try (the CPU outside, a job's weight), and see what would start, with how many cores, and why the rest wait. It reads no config and no queue.
Breaking changes
status --jsonis format 2. New:state_since_ms;busy_beyond_running_jobsunrounded, andcpugiven whenever the CPU outside the queue was measured;other_queuesas{file, readable, running, waiting, parked}. Within format 2, fields may be added (ignore the ones you do not know); anything else, including a new value ofstate,needorreason, will be a new format. See the README, "Status as JSON".status's FOR of a resumed job counts from when it resumed, and that of an orphaned job is unknown (-), not from when it started.runandstatusnow advise 0.12 or later for every agent, rather than one version.
Install
mise use -g github:ktanaka101/loadmaster@0.14.0Or download the archive for your platform below: macOS (Apple Silicon and Intel) and Linux (x86_64 and arm64, statically linked).
v0.13.0
Waiting costs next to nothing now: a poll that finds the queue unchanged keeps its last decision, and jobs of a kind are judged once, so 160 waiting jobs take 0.1 cores in all instead of 4. Where the jobs record the same limits, the rule now starts every job that can start without risking one.
Changed
-
Waiting jobs use far less CPU. The rule depends only on the jobs and the CPU outside the queue, so a poll that finds the database as the last one left it keeps the last decision (whether the jobs are alive, and how old the CPU measurement is, are still checked on every poll). And two jobs with the same amounts and limits have the same condition, so one start stands for every later one of its kind: judging a job costs as many steps as jobs times kinds, not pairs of jobs. Behind one job holding an 8-core machine, the waiting processes used in all:
Waiting jobs 0.12.0 0.13.0 40 0.14 cores 0.006 cores 80 0.65 cores 0.018 cores 160 4.05 cores 0.10 cores -
No job waits that could start without risking a limit, where the jobs record the same limits and start with their weights. Jobs may end and leave at any time, so any set of the jobs ahead that fits within its members' limits may be what runs when a job ahead starts; the rule now checks exactly that, and a test compares it with a search of every such set. Before, a dimension that could not bind on its own (often CPU) was counted at its sum even where the sets that fit on another one (a resource) could not use all of it: 0.1% of the decisions in a random simulation, 1.3 to 2.5% of the mean wait.
-
A job that cannot run beside a start by its own limit no longer holds others back. Where repositories record different capacities, a running job whose smaller capacity rules out a waiting job's start beside it was still counted beside that start.
-
The rule changes were reviewed by Codex (GPT-6-Astra). What it found in versions before this one, among them one that let a job past a limit and one that made
statuscubic in the line, is fixed and covered by tests.
The decisions are otherwise the same, and so are the promises: no job is delayed by jobs that got in line after it, and the running jobs never use more than the smallest limit any of them records.
Upgrading
The queue file stays queue-v4.db, and 0.12 and 0.13 keep the limits together in it, so agents can move to 0.13 one at a time.
mise use -g github:ktanaka101/loadmaster@0.13.0Or download the archive for your platform below: macOS (Apple Silicon and Intel) and Linux (x86_64 and arm64, statically linked).
v0.12.0
A job now starts if it fits now and will still fit beside every job ahead of it when that one starts: waiting jobs no longer set aside cores and resources for later, so checks run in room that set-asides used to keep empty, with the same promise that no job is delayed by jobs that got in line after it. status --json gives tools a documented format.
Changed
- The admission rule. A job decides from the jobs ahead of it alone, so no job is delayed by the jobs behind it; what the rule protects is the limits. A job may start if (1) it fits now beside the running jobs ahead of it and those about to start, and (2) for every job ahead of it that waits, it will still fit beside that job when it starts, next to the most the other jobs ahead could use beside it then. Set-asides, "counts on", and a job short of CPU holding back everyone behind it are gone: all of it follows from (2).
- Unit runs waiting in turn for the browser a long UI capture holds keep no cores from the checks behind them, however many wait; a second capture waiting behind the first does not hold back the cores left beside it.
- With CPU alone, a small job may start ahead of a big one that waits, where it cannot be in its way when that one starts. When the big one needs the whole machine, the small one still waits.
- Running jobs never use more, on any dimension, than the smallest limit any of them records (capacities that repositories or machines record differently). CPU busy outside the queue only throttles starts, as before: jobs that started earlier may keep the machine above the lowered limit, never above the capacity. Measure-only jobs start whatever the rule says, so the promises do not hold while one runs or after one started.
- The rule was reviewed three times; each counterexample found is a unit test, and the simulations add departures, parked places that resume or expire, jobs that adapt to CPU outside or not, disagreeing limits, and random poll orders.
- Every use of a resource counts in its checkout. A use without a per-checkout limit of its own was left out of its checkout's count, so a run could take a checkout past the limit another task recorded. Fixed.
- Why a job waits (
status, the waiting messages): either it does not fit now (3 cores (7 of 8 in use)), or it would not fit beside a start ahead (3 cores (6 of 8 could be in use when #43 `e2e` starts, beside #41 `test`)), naming the job ahead whose record set a smaller limit when there is one. "Sets aside" and "counts on" are gone. loadmaster checkreports a waiting run as keeping a task out only if the task fits beside the long run the waiting run waits for, but not beside the waiting run's start, and would if the waiting run took one core; the rule itself decides it.
Added
loadmaster status --json: the queue for tools, in a versioned format documented in the README ("Status as JSON"): capacity, resources, CPU outside the queue, other versions' queues, and per job its state (running,waiting,parked,orphaned,lingering), amounts, limits, processes and why it waits, as data and as the sentence the table shows. Any change to the format will be a breaking change of a release; the table may change in any release.
Breaking changes
- The queue moved to
queue-v4.db. Its schema did not change, but the admission rule is the protocol between the processes that share a queue, and 0.11's rule does not protect the new promises. Upgrade every agent on a machine at once (mise use -g github:ktanaka101/loadmaster@0.12.0); until then,statusandrunsay how many jobs run in the old queue, which the new one does not count. Deletequeue-v3.db(with-waland-shm) in~/.local/state/loadmasteronce no 0.9 to 0.11 job runs. - The reasons in
statusand the waiting messages are worded differently (see above).
Install
mise use -g github:ktanaka101/loadmaster@0.12.0Or download the archive for your platform below: macOS (Apple Silicon and Intel) and Linux (x86_64 and arm64, statically linked).
v0.11.0
A job waiting for what a running job holds no longer sets aside what that job frees when it ends, so the checks behind it run meanwhile; and status tells a job whose command exited, leaving processes that hold its slot, from one whose loadmaster is gone.
Changed
- A waiting job sets aside only what it is not sure to get back. A set-aside is there so that a waiting job finds what it needs free once it can start, so it now sets aside its amount less what is sure to be freed by then. When, on every dimension it is short of, the running jobs ahead leave it no room by its own limit, only the end of a job ahead that holds such a dimension can let it in, and that end frees what the job held: the waiting job counts on it, and jobs behind may use that much meanwhile. What one end frees is counted on once. When something that frees nothing could let it in instead (a job ahead leaving the line, CPU outside the queue going down), it sets aside everything, as before. No job is delayed by jobs that got in line after it, as before.
- A unit run waiting for the browser that a long UI capture holds used to set aside its cores, and a lint behind it, short of those, held back every check behind it. The unit run can only start once the capture ends, which frees the capture's cores: it now sets none aside, and the lint starts.
- A job waiting for its checkout's limit no longer sets aside the machine-wide resource that the run ahead in its checkout frees, so a job from another checkout takes the one that is free.
loadmaster statusand the waiting messages say what a waiting job counts on, such ascounts on 2 cores that #1 `ui` frees when it ends, besides what it sets aside.loadmaster checkwarns about weights by the same rule.
- Jobs whose command exited are listed as lingering. A task whose command exits while processes it started in its process group still run keeps its slot until they exit.
loadmaster statusused to list that job as orphaned, saying loadmaster was gone; it now lists it as lingering, with the processes in its group, andloadmaster cancel IDstops them. Orphaned now means only a job whose loadmaster is gone while its task's command still runs. The message at the handover names the processes too.
Upgrading
The queue's layout is unchanged: 0.10 and 0.11 share queue-v3.db. Each job is admitted by its own loadmaster, and 0.10's rule sets aside more, so the two can run side by side; a job gets the new rule once its agent runs 0.11.
Install
mise use -g github:ktanaka101/loadmaster@0.11.0Or download the archive for your platform below: macOS (Apple Silicon and Intel) and Linux (x86_64 and arm64, statically linked).