feat: let a job give up when it waits too long in the queue

[stephenc]

What it does

--timeout limits the time that a job runs. --max-queue-time limits the
time that a job waits before it starts.

ID=$(qex submit --max-queue-time 30m -- make test)
qex wait $ID          # this gives an answer inside 30 minutes

The option is a qex submit option, a job file field (max_queue_time), a
pipeline stage field, and [defaults] max_queue_time in the config file. It is
plumbed in the same shape as timeout: cli.rs -> spec.rs -> the job record
-> sched.rs.

This closes part of issue #10 ("Smaller, agreed": --max-queue-time, so a job
that never starts gives up and says so).

The decisions, and why

A new state, expired. Not timeout. A job with timeout ran, used the
machine and wrote output; a job with expired never got the machine and its log
file is empty. The reader's next action is different for each: make the work
faster, against give the job a smaller claim or wait for a quiet machine. A
reader who cannot separate the two opens an empty log file and learns nothing.

A new exit code, 123. Not 125 (kill, timeout, out-of-memory) and not 126 (a
job that it needed failed). A script can thus separate "the work is too slow"
from "the machine never had capacity". The existing contract stays: 124 is still
the timeout command code for the wait itself, and --passthrough gives 123 as
well, because a job that never ran has no exit code of its own.

The reason text. The record says what the job waited for and how long:

expired  the job did not start. It waited 5s in the queue, and its
         --max-queue-time is 5s. The last reason was: the job claims 64 cores
         and the budget is 2 cores; ... Give the job a smaller claim, wait until
         the machine is quiet, or give a longer --max-queue-time, then submit
         the job again.

No built-in default. A job that qex discards is work that a person wanted, so
a default value would throw away the work of a user who never asked for the rule,
on the day that the machine is busy. [defaults] max_queue_time gives the rule
to every job of a user who does ask, one time.

The clock starts at submitted_at. Not at the last scheduling pass. A
coordinator that stops and starts again continues the same count. Without that,
a restart would give each queued job a new full wait and the limit would promise
nothing.

A wait for a dependency counts. The option answers one question for the
reader: does this id give an answer inside this time? A clock that stopped while
a job waited for needs could not answer it, because a chain of slow stages
would hold the clock for hours. The documentation says so, and it says the
remedy: give a stage a value that covers the whole pipeline, or give it none.

The capability max-queue-time. A coordinator without it would keep the
field, give a job id, and let the job wait for ever — exactly the fault that the
option removes. The CLI refuses such a job and names the option.

The race that must not come back

A scheduling pass chooses a job, releases the lock, and start_job takes the
lock again and writes starting. Two guards stop a pass from writing expired
over a job that runs:

1. the job that the pass chose is never in the list of jobs that expire, so a
   job that can start starts even in the second when its limit ends;
2. expire refuses a job whose state is not queued, which covers the moment
   between the two locks.

What I tested

New unit tests: the three sources of the value and --max-queue-time 0; no
limit by default; a job past its limit expires with a reason that names the
wait; a job inside its limit and a job with no limit stay; a job that started
never expires (both guards); a coordinator that cannot obey the option refuses
the job.

New end-to-end tests: a_job_that_never_starts_gives_up_and_says_why (a claim
larger than the budget with oversized = "queue", so the job can never start:
it becomes expired with no start time, no exit code, and a reason that names
the budget, and qex wait gives 123) and
a_job_that_started_at_its_queue_limit_keeps_its_result (ten jobs on a budget of
one core with a 2s limit, so the start and the limit meet: a record that says
expired must hold no start time and no exit code).

What I measured

cargo fmt --all                          clean
cargo clippy --all-targets -- -D warnings clean
cargo test --bins                        178 passed, 0 failed (0.08s)
cargo test --test e2e -- --test-threads=2 73 passed, 0 failed (49.11s)

A manual run against a temporary state directory gave the record above, the same
text in qex list, qex status and qex wait, and the exit code 123.

Cargo.toml moves from 0.7.1 to 0.8.0.

🤖 Generated with Claude Code

https://claude.ai/code/session_01KNvagiMEU3myn8EXGaGEM9