fix: start the jobs behind a job that another user holds back

[stephenc]

Closes part of #10 — the head-of-line half. It implements section 1 of
design-decisions.md.

The rule, in one paragraph

qex walks the queue in order. The first job that cannot start is the job at the
front, and the class of the holder of the capacity decides what happens behind
it. If the jobs of this queue hold the capacity, or if the job is larger than
the budget and waits for a quiet machine, qex schedules that release: two jobs
may pass ([queue] max_bypass, default 2), and after that qex keeps the
capacity and starts nothing at all. If another user, or a program outside qex,
holds the capacity, qex does not schedule that release, so the job at the front
never keeps capacity and every job behind it starts — an empty machine gives
such a job nothing, and to hold the machine for it is the measured fault. The
count of the jobs that passed is not reset when the class changes, so a job
that another user held for an hour is unpassable in the same scheduler cycle in
which a job of this queue becomes the holder.

What changed

• Admit::No carries a Blocker class: Sibling, Peer { count },
  Machine, OversizedWaitsForIdle, OversizedParked.
• Two new status.json fields: blocked_since and passed_by. Both are in
  qex schema status.
• New [queue] max_bypass, default 2. max_bypass = 0 gives the exact strict
  order of 0.7.1.
• The reservation is total: when the job at the front is unpassable, qex starts
  nothing. This covers every resource with no extra bookkeeping, which is what
  the pool feature (question 2) leans on.
• A lock shortage keeps its continue. A job that waits for a lock never
  becomes the job at the front and never keeps capacity.
• Every job that waits gets a reason from its own admit call. The earlier loop
  stopped at the first job that could not start.
• qex info and qex top give one queue-health line. The new Response::Info
  fields are Option, and qex info --json gives null for unknown.

The two performance traps

• /proc on every call. admit now runs for every ready job in each cycle.
  A Machine record takes the available memory, the memory pressure and the
  peer claims one time for each choose call. This removes the per-cycle
  per-job storm of system calls, and it also stops two jobs in one pass from
  getting answers from two different moments.
• A reason change writes the record with two fsyncs. The bypass count is in
  its own field and never in the reason text. The one sentence that holds a
  count — the sentence that a job behind an unpassable job reads — cannot move
  while a reader sees it, because no job starts while the capacity is kept.
  choose also collects the changed records and writes each one once.

Decisions where the document left something open

1. The peer claims are cached with the machine measurements. §4.2 names
   /proc only, but peers::claims reads a directory of files on every
   admit call, and that is the same fault. It is in the same one-per-pass
   record.
2. queue_state is the mark of a coordinator that reports the health.
   §4.1 asks for one machine-readable enum plus one human sentence, and §4.2
   asks for options, not defaulted scalars. last_start_at: Option<u64> alone
   is ambiguous: None would mean both "no job started" and "the coordinator
   is too old". queue_state is None only in the second case, so a reader
   tests one field. The words are running, held, waits-for-peer,
   waits-for-machine, waits-for-capacity, waits-for-idle and parked.
   The last three are additions to the list in §4.1, which did not cover a
   sibling wait or an oversized job.
3. The human sentence is composed in the CLI, in one function that qex info and qex top share. A second wire field for the sentence would
   drift from the fields it describes. The function gives unknown for each
   None.
4. OversizedWaitsForIdle gained a text for the case in which it keeps the
   capacity ("qex starts no other job before this one, so the queue becomes
   empty"). §1.3 gave no wording for it.
5. paused, paused_at and paused_reason are not in this PR. They
   belong to question 3, which another branch implements, and adding them here
   would give two branches that write the same fields.

Where I disagree with the document

§1.2 and §4.3 say that a job which a peer blocks "becomes unpassable in the
same tick that the peer releases". A job cannot go from Peer to a blocked
Sibling on a release alone: the sibling test uses this queue's own claims
only, so a test that passed before the release passes after it, and the job
starts. The transition to Sibling happens when a job of this queue takes the
capacity, and that is where the carry-over earns its value: without it, that
change of holder would reset the count and a stream of small jobs would pass
the job at the front for ever. The release of the other user must also not
reset the count. The test asserts both, and it is named
a_job_blocked_by_a_peer_becomes_unpassable_when_the_holder_changes. The
mechanism the document describes is the one that is implemented; only the
sentence about "the same tick as the release" is not reachable.

What I measured

On this machine, with [budget] cpu = 4 and another coordinator that holds 3
cores:

• Before: a job of 4 cores at the front kept every job behind it in the queue,
  and each of those jobs read waits for the job <id> at the front of the queue.
• After: the jobs behind it complete, and the job at the front reads this job cannot fit while another user holds capacity: the job needs 4 cores, this queue holds 0 of the 4 cores in the budget, and 1 other user holds 3 cores. qex does not control that user, so this wait has no known end. ...
• qex info gives queue: waits for another user · ... · 1 other user holds 3 cores and 64MB · the job at the front is <id> (true).
• With max_bypass = 2 and a stream of six small jobs behind a large sibling
  job: exactly two pass, and the queue is then held.
• With max_bypass = 0: none pass, and passed_by stays 0.

What I tested

New unit tests in src/sched.rs:

• a_job_that_another_user_holds_back_says_so_and_never_keeps_capacity
• the_queue_keeps_capacity_only_for_a_holder_that_it_schedules
• a_job_that_the_config_parks_does_not_stop_the_jobs_behind_it
• a_job_that_waits_for_a_quiet_machine_keeps_the_capacity
• a_sibling_wait_says_if_another_job_can_pass_it

New end-to-end tests in tests/e2e.rs, each of which starts two coordinators
that share a peer directory where it needs another user:

• a_job_that_another_user_holds_back_does_not_park_the_jobs_behind_it — the
  measured fault.
• a_job_behind_a_job_that_keeps_no_capacity_gets_its_own_reason
• a_job_blocked_by_a_peer_becomes_unpassable_when_the_holder_changes — the
  carry-over rule.
• a_stream_of_small_jobs_does_not_pass_a_large_job_for_ever — starvation is
  still prevented.
• max_bypass_zero_lets_no_job_pass_the_front_of_the_queue
• a_job_that_the_config_parks_does_not_park_the_jobs_behind_it — the second
  head-of-line path, with oversized = "queue".

Results:

cargo fmt --all                            clean
cargo clippy --all-targets -- -D warnings  clean
cargo test --bins                          177 passed, 0 failed
cargo test --test e2e -- --test-threads=2  77 passed, 0 failed

Also updated: docs/reference.md, qex help config, qex help resources,
src/schema.rs, and Cargo.toml to 0.7.2.

🤖 Generated with Claude Code

https://claude.ai/code/session_01KNvagiMEU3myn8EXGaGEM9