The agreed backlog

[stephenc]

The features that are agreed and not yet written, so that none is lost. Each
gets its own branch and its own pull request.

Pause and resume has a page of its own: #9.

From an independent review of the tool

A group is not a handle. qex pipeline writes a group id to stdout — the
obvious thing to keep — and then qex wait $G, qex status $G, qex kill $G
and qex clean $G all answer "there is no job with the id ..." with the code
127. Only qex list --group takes it. The documented way to use a pipeline thus
ends with the user finding the last stage by hand. 127 is also the code for "you
gave nonsense", so a script cannot tell the two apart. Names already resolve
everywhere, so the resolver exists; it does not know about groups.

An out-of-memory kill teaches nothing and never retries. --retries fires
on the state failed only, and an out-of-memory kill gives the state killed.
The learner records the jobs that COMPLETED only. So the case in the README —
a long training run with --mem guess that the kernel stops at hour four —
records nothing, and the same claim dies the same way on the next run. It should
retry with the claim doubled and say that the claim was too small.

A job that is too large stops the queue behind it. Measured: one job that
could not be admitted, because a peer held part of the budget, parked two small
jobs behind it for ever. The rule is deliberate and the words are not: the
reason says "waits for the job at the front", and a user sees a queue that never
moves with no cause. Whatever the rule becomes, the reason must say "this job
cannot fit while another user holds capacity".

Nothing makes a submission idempotent. The agent page tells an agent that
comes back to a session to run qex list and look. That is a proxy check, which
is the fault this tool exists to remove. qex submit --dedupe-key build:$(pwd)
should give the id of the job that already operates and exit 0.

Countable resources, GPUs and VRAM

The scheduler counts two things, cores and memory, and holds named locks. A
machine with 4 GPUs has no way to say --gpu 1.

They must be PROMISES in the same way as cores and memory, so that a machine
with no CUDA and no driver library still schedules them: a count in the
configuration, a claim on the job, and the same arithmetic that admits a job
today. Assignment comes after that — the coordinator tells the supervisor which
index it gave, and the supervisor writes CUDA_VISIBLE_DEVICES — and it is what
makes the promise useful. VRAM is a quantity, in the same shape as memory.

Counted locks give the same thing for everything else: --lock net:1 against a
pool of 4 means "at most four jobs that use the network".

A limit on the output of a job

A job wrote 386MB of standard output in a review and nothing stopped it. qex is
made to be started and left, and that is exactly when nobody sees a disk fill —
the same disk that holds status.json. [logs] max_bytes, keeping the head and
the tail with a line that says how much went.

An event stream

qex events --json, a line for each change of state. An agent that drives
twenty jobs reads one stream instead of asking about each job. The socket
protocol exists and this is one more request name.

Smaller, agreed

• A hook when a job stops, in the configuration and not on the job.
• --max-queue-time, so a job that never starts gives up and says so.
• Fan-out: qex submit --each-line inputs.txt -- ./process {}, one group.