feat: let a person pause the queue, or take a lock, and say so everywhere

[stephenc]

Closes #9.

What it does

qex pause queue [--reason TEXT] [--for TIME] [--drain]
qex resume queue
qex pause lock <name>
qex resume lock <name>
qex pause                  # say what is paused now

qex resume with no word starts the queue again, so every message that names
that command is correct in its short form too.

Pause the queue. qex starts NO job. The jobs that operate continue, because
each one already holds its capacity and a stop would lose that work; the answer
says so and points at qex kill and qex pause queue --drain. --drain gives
control back when no job of this queue operates.

Pause a lock. qex pause lock gpu0 gives the lock to the person. Every job
that needs it waits, with the reason waits for the lock gpu0, which a person holds. The command is never refused while a job holds the lock: qex
records the request, that job keeps the lock, no other job takes it, and the
lock comes to the person when the job stops.

It survives a coordinator that stops. The state is paused.json in the
runtime directory, beside the job records. The coordinator writes it and reads
it in recover, so it has one writer.

It says so. qex info, qex top, qex list and qex status each report
the pause, how long it has lasted, and — loudly — when it has no end. qex submit warns on stderr at the moment of the submission.

What I decided, and why

• A paused queue starts nothing, per section 3 of the design document. No
  exception for a job that claims little: every job in qex has a claim, the
  point is a quiet machine, and paused must be one fact an agent can act on
  rather than a predicate it must qualify.
• The pause reason holds a clock time and no elapsed time. Section 3.3 gives
  the text "… paused it at 14:32, 6 minutes ago". Section 4.2 point 2 forbids a
  changing number in blocked_reason: the scheduler writes status.json with
  two fsync calls for each reason that changes, and it ticks every 500ms, so
  "6 minutes ago" would rewrite the record of every job in the queue twice a
  second for the whole length of the pause. The clock time is stable, and the
  elapsed time is calculated by qex info and qex top when a person reads
  them. A unit test holds this rule.
• Info gains queue_state, not a paused boolean. Section 4.1 asks for
  one machine-readable field, because two parallel fields drift. It is
  Option<String> and not a Rust enum, so a later version can add held,
  waits-for-peer and waits-for-machine (question 1) without making an
  earlier CLI refuse the whole answer. None prints unknown, never
  running — a guess here is a lie in the one place where the honest answer
  matters most.
• Messages name qex resume queue, the command that the user types, in
  place of the qex resume of section 3.3. Both forms operate.
• Locks accept --for and --reason too. One shape for both targets, and
  a lock that a person forgets has the same fault as a queue that a person
  forgets.

The four consequences of section 3.4

1. The dependency pass still runs while paused. The pause test is between pass 1
   and pass 2 of sched::choose, so a job whose dependency failed still becomes
   skipped and qex wait still gives an answer.
2. The pause test is before the oversized branch. A paused queue is idle by
   construction, so without this a pause would start every oversized job.
3. qex resume sets idle_since to now. Without it the settle window is
   already satisfied at the resume and the first job to run would be an
   oversized one, alone, in front of everything that waited. A pause that ends
   by itself (--for) does the same.
4. The pause is a file, read in recover.

Capability

pause is in capabilities::ALL, and capabilities::require is the second
entry point beside required_by: this is a request name, not a job field. An
earlier coordinator answered "qex could not read this request", which states a
condition and gives no remedy, and the failure is the dangerous one — the person
believes that the machine is quiet while the queue starts jobs.

No JSON schema changed: qex schema job|status|pipeline cover the job file and
status.json, and this feature adds no field to either.

What I measured

• cargo test --bins: 179 passed, 0 failed (7 new).
• cargo test --test e2e -- --test-threads=2: 77 passed, 0 failed, 58.5s
  (6 new).
• cargo fmt --all and cargo clippy --all-targets -- -D warnings: clean.
• By hand, on an isolated state directory: qex pause queue --reason, qex info, qex list, qex pause, qex pause lock gpu0, qex resume, qex resume lock gpu0. paused.json appears at the first pause and is deleted
  when nothing is paused.

What I tested

• a_paused_queue_starts_no_job_and_the_jobs_that_operate_continue
• the_pause_survives_a_coordinator_that_stops — takes the pid from
  qex info --no-start --json, never a process-list search, kills that
  coordinator, lets the next command start a new one, and confirms
  queue_state and paused_reason on the NEW pid.
• a_person_gets_a_lock_when_the_job_that_holds_it_stops
• a_job_that_waits_for_a_pause_says_the_pause — and not "waits for …"
• a_pause_with_a_time_ends_by_itself
• a_failed_dependency_is_still_skipped_while_the_queue_is_paused
• Unit tests: the expiry arithmetic, the JSON round trip, the stable reason
  text, the loud "NO END" line, the qex top header, and the capability
  refusal.

Each test that measures "nothing started" measures over a period, and not one
time, so a job that starts late cannot pass it.

Documentation

docs/reference.md gains a section and the command list; qex help pause is a
new topic (qex help resume is an alias); Cargo.toml moves to 0.8.0.

🤖 Generated with Claude Code

https://claude.ai/code/session_01KNvagiMEU3myn8EXGaGEM9

[stephenc]

Review addressed in 561320d. Both blocking holes are closed, and all six non-blocking items are done.

BLOCKING 1 — a retry started work behind the pause. supervisor::main now calls wait_while_paused before return main(id). It reads Paused::read(), expires it, and waits while the queue is paused OR while a lock of that job is paused, writing the pause as blocked_reason once — the text holds no number that changes, so a second write would be the same bytes and two more fsync calls twice a second. I fixed the lock half in the same change: the message stayed truthful, but the machine did not, and the two share one line of code. The module doc of src/pause.rs now records that "a file is a state that every process can read" is the second reason for the file. I did not move the recursion or re-check the budget — that is the pre-existing main bypass and belongs with #22.

BLOCKING 2 — an unreadable paused.json meant "not paused". It now holds the queue, which is the option you called better. Paused::read separates the three cases: no file → nothing paused (the usual state); file present but unreadable or unparseable → a PauseRecord { fault: true } whose reason names the path and the parse error. queue_reason and queue_line branch on fault, because "A person or an agent paused it at 14:32" would be false — no person asked. recover logs it and writes the record back, so the next command reads a file it can parse and the words stay with it. Info.queue_state gains paused-by-fault — a third state of the one field you approved, not a flag beside paused, so nothing drifts. qex resume queue writes a fresh file and recovers. Unknown fields still parse (unit test).

Your framing was right and my doc comment was wrong: "qex must not invent a pause" is a good rule for a tool that reports, and the wrong rule for the tool that holds the machine.

3 — start_job now re-tests the pause and the job's locks under the state lock, and the comment block at :567 names why. 4 — pause_report expires before printing. 5 — keep_the_end keeps paused_at, and keeps the until/reason that the second command does not give; a fault record is replaced in full. 6 — capabilities::require takes the danger as a parameter; resume says "That coordinator does not read the pause record, so it already starts the jobs of the queue. This command would change nothing." 7 — --for 0 errors and names qex resume queue. 8 — the CLI sends by_pid in the Pause request (verified: the record holds the CLI pid, not the coordinator's).

Tests. cargo fmt --all and cargo clippy --all-targets -- -D warnings clean. cargo test --bins: 182 passed, 0 failed (+3). cargo test --test e2e -- --test-threads=2: 82 passed, 0 failed, 87.17s (+5): a_retry_starts_no_new_attempt_while_the_queue_is_paused, a_retry_does_not_take_a_lock_that_a_person_holds, a_pause_record_that_qex_cannot_read_holds_the_queue, a_pause_for_zero_is_refused, a_second_pause_keeps_the_end_of_the_first. The two retry tests count attempt marks in stdout.log over a 6s period, not once, so an attempt that waits out the retry's one-second sleep cannot pass. qex help pause and docs/reference.md carry the new rules.

The qex wait item: yes, file it. qex run reads blocked_reason and qex wait does not, so the command an agent is told to use is the one that goes silent — and a pause makes that silence long. It is pre-existing and belongs in its own PR.