feat: raise the claim and run the job again after a kill for memory

[stephenc]

Closes part of #10 — "An out-of-memory kill teaches nothing and never retries".

What the feature does

The headline case in the README: a long run with --mem guess that the kernel
stops at hour four. Before this change, that job got the state oom,
--retries did not see it (it fires on failed only), and the learner recorded
completed jobs only. The claim was never corrected, and the next run died the
same way.

Now:

1. The kernel stops the job → qex multiplies the claim, starts the job again
   with the same id and the same record, and the log holds every attempt.
2. The record says plainly what happened:

state:     completed
claim:     1 core(s), 256MB  (qex raised it, because the earlier claim was too small)
note:      the kernel stopped attempt 1 of this job, because the job used more
           memory than its claim of 128MB. THE CLAIM WAS TOO SMALL. qex raised
           the claim to 256MB and starts the job again.
attempts:  2

3. The learner keeps the lesson as a lower bound, so the next run of the
   same command starts above the claim that failed.

The decisions, and why

How qex tells a kill for memory from every other kill. The source is the
cgroup counter memory.events oom_kill, read before and after the job. It was
already there and it needs no limit from qex: every process on Linux is in a
cgroup, so the count exists in the usual configuration.

That counter alone is not sufficient, and this is the part that could do harm.
When qex applies no limit it reads the count of the SESSION, which also counts a
kill in a different program of the same user. So qex kill now writes a mark
in the job directory before it sends the signal, and that mark always wins. A
job that a person stopped can never be answered with a larger claim and a new
attempt — qex must not repeat work that somebody stopped on purpose.

When qex cannot tell. macOS has no cgroup and no equivalent count. qex then
gives the state killed, which is the safe answer because it starts no new
attempt, and it writes a note in the record saying that it could not tell the
cause and what the reader should compare. A guess in either direction sends the
reader to the wrong correction.

Its own count, not --retries. --retries is for a fault outside the task,
and the user chose that number for that fault. A claim is usually the work of
qex (--mem guess and the learned claim both come from qex), so qex corrects
its own fault and does not spend a budget given for something else. A job with
no --retries value — the README case — still gets the correction.
[retry] on_oom (default 2) and [retry] growth (default 2.0) control it, and
on_oom = 0 turns it off.

The bound. Two rules stop the ladder:

• a number of raises: 2 raises give 4 times the first claim, which corrects the
  usual error of an estimate. Each attempt costs the full time of the job — four
  hours in the README case — so a ladder with no limit can use a day of the
  machine and give no result.
• the memory budget: the claim never goes above [budget] mem. qex must not
  claim memory that it does not have. A job that already claims the whole budget
  keeps the state oom, and the record says to use a larger machine or a larger
  budget.

Where the raised claim lives. In status.json (mem), and not in
spec.json. The specification keeps what the user asked for; the record gives
the claim in force. The coordinator budgets on status.mem and re-reads that
file every 500ms, so the raised claim reaches the budget accounting on its own,
and the memory limit of the next attempt now comes from the record as well
(before, it came from the specification, so a cgroup limit would have stopped
every new attempt at the size that already failed).

The learner. A sample now has a kind: peak (the job completed, so this
is the memory the job needs) or lower-bound (the kernel stopped the job, so
the true need is ABOVE this value). A missing kind reads as peak, so an
existing usage.json keeps its meaning exactly and no claim moves for a job
that behaves. suggest takes the largest peak AND the largest lower bound and
gives a claim above both, so three later small runs cannot average away a
measurement that cost a whole run. The lower bound is
max(measured peak, claim): with a limit the kernel stops the job at the claim,
and with no limit the peak is the evidence qex has.

What I measured

• [retry] on_oom = 2, growth = 2.0: 128MB → 256MB → 512MB, three attempts.
• Learner: after a kill at a 128MB claim, the next submission of the same
  command with no --mem at all got 192MB (the bound plus the 1.5 margin)
  with claim_source = learned.
• qex config show gains one section; qex status --json gains one field.

What is tested, and which kills are real

No test makes a genuine kernel kill, and I did not write one that pretends
to. A genuine kill needs a cgroup memory limit or a machine with no free
memory. qex can apply a limit only when the coordinator owns its cgroup, and a
usual machine gives the login session to the root user (availability() is
Unavailable here); filling the machine is not acceptable when other work runs
on it, and the kernel chooses its own victim. This is stated in a doc comment on
the test helper.

The e2e tests instead make the evidence that a true kill leaves — the
out-of-memory record in the job directory — and the job then stops itself with
SIGKILL, the same signal the kernel uses. Every step after that point is the
real code: the classification, the new claim, the new attempt, the words in the
record, the log of every attempt, and the learner. The reading of the cgroup
counter itself is unchanged code, covered by the existing enforce tests.

Test	Kind
a_job_that_the_kernel_stops_for_memory_runs_again_with_a_larger_claim	e2e, simulated kill
a_job_that_a_user_killed_is_not_retried_and_teaches_the_learner_nothing	e2e, real qex kill + planted counter evidence
a_claim_that_stays_too_small_stops_at_the_limit	e2e, simulated kill
supervisor::a_kill_for_memory_gives_the_state_oom	unit
supervisor::a_job_that_a_user_stopped_is_never_an_out_of_memory_kill	unit
supervisor::a_kill_that_qex_cannot_explain_says_so	unit
supervisor::the_claim_doubles_after_a_kill_for_memory	unit
supervisor::the_claim_stops_growing_at_the_limit	unit
supervisor::the_claim_never_goes_above_the_memory_budget	unit
supervisor::the_config_file_can_stop_the_correction	unit
usage::the_next_claim_is_above_a_lower_bound	unit
usage::a_lower_bound_is_not_averaged_away_by_the_smaller_runs	unit
usage::the_largest_evidence_wins_whatever_its_kind	unit
usage::an_earlier_file_keeps_its_meaning	unit
usage::a_kill_for_memory_records_the_claim_when_it_is_larger	unit

The user-kill test is the one that protects the feature from doing harm, and it
is falsifiable: with mark_user_kill removed, it fails — the job is classified
oom and qex starts it again with a doubled claim. That is exactly the harm the
mark prevents.

Results:

cargo fmt --all                            clean
cargo clippy --all-targets -- -D warnings  clean
cargo test --bins                          184 passed, 0 failed
cargo test --test e2e -- --test-threads=2   74 passed, 0 failed  (47s)

Also in this change

• docs/reference.md: the two kinds of measurement, and a section on a job that
  the kernel stops for memory.
• docs/design.md: what the learner records.
• README.md: the case, with the record that a user reads.
• src/help.rs: the retries option, the learning topic, the states topic, and
  the config topic.
• src/schema.rs: oom_raises, and mem, claim_source, attempts and
  retries_left, which the status schema did not name.
• Cargo.toml: 0.7.1 → 0.8.0.

🤖 Generated with Claude Code

https://claude.ai/code/session_01KNvagiMEU3myn8EXGaGEM9

[stephenc]

Review fixes — both blocking findings corrected

Pushed as 5844aa4. Version 0.8.0 → 0.8.1 (a second local build must not report the number of the first).

BLOCKING 1 — the raised claim now goes through admission

The supervisor no longer starts the new attempt. It writes the record with state: queued and the new claim, and it exits. reap gained a branch for a non-terminal record: it puts the job in the queue again (with State::enqueue, the same priority rule as a submission) and the scheduler admits it against the budget like any other job.

The spec.mem / status.mem split is gone: size_check and admit now take the claim in force (status.cpu, status.mem) instead of borrowing the spec, so the budget, the cgroup limit and the admission all read one number. size_check(cfg, cpu, mem) and admit(cfg, cpu, mem, used…) take plain values now; the submit-time call in daemon.rs passes spec.cpu, spec.mem, which is correct there because no record exists yet.

The --retries recursion stays in the supervisor, exactly as you noted: it does not change the claim, so its accounting is already consistent.

Falsification: I restored return main(id) in place of the surrender and re-ran a_raised_claim_waits_for_capacity_in_the_queue — it failed with left: "completed" right: "queued", the job having run at 1GB while the holder held 400MB of a 1GB budget. Exactly your reproduction.

BLOCKING 2 — qex now acts only on job-scoped evidence

New enforce::OomScope { Job, Session }, written into the oom record:

[enforce] mode	qex reads	qex does
soft / hard	the cgroup qex made for this job	report oom, raise, re-run, teach the learner
off (default)	the cgroup of the login session	report oom, say what to do, no new attempt, no learner sample

mark_oom keeps the stronger evidence if both tests fire. An oom file from an older qex holds 1 and reads as Session — the safe value.

The session-scope record says: "qex applies no memory limit to a job in this configuration, so it counts the kills of the whole login session and it cannot prove that the claim of this job was too small: the machine can be full while the claim is correct. qex therefore did NOT start the job again… set [enforce] mode in the config file, and qex then corrects the claim itself."

Falsification: with the scope gate disabled, a_kill_for_memory_with_no_limit_reports_but_does_not_run_the_job_again failed with attempts: 3, oom_raises: 2, claim_source: raised — your reproduction.

This does mean the correction is off in the default configuration. That is the honest position: without a per-job cgroup, qex has no evidence that the claim was the cause. The docs, the help and the README now state it plainly rather than implying the feature always fires.

NON-BLOCKING 3 — the "lower bound" is now evidence

usage::record handles completed only; usage::record_lower_bound is a separate function whose doc comment states that the caller must hold job-scoped evidence. Under that gate max(peak, claim) is exactly right — the kernel stopped the job at the limit qex set from the claim. Two further changes for the ratchet you found:

• One lower bound per command. A ladder made one per step and filled three of five slots. The largest bound holds every fact the smaller ones do, so qex keeps that one (moved to the end so it stays newest).
• A learned claim never exceeds [budget] mem, capped in spec.rs. qex makes that number itself and must not make one it then refuses with "Decrease the claim" when the user gave no claim. A job that truly needs more now runs and gets the actionable message ("qex has no larger claim to give. Use a machine with more memory").

NON-BLOCKING 4, 5, 6, 7

• 4. clear_user_kill, called with clear_oom at the top of main() — before status.pid is set, so no qex kill can have marked yet. E2E a_mark_from_one_attempt_does_not_decide_the_next_attempt reproduces your sequence (TERM → exit 3 → failed → retry → OOM) and asserts oom_raises: 1 and retries_left: 1.
• 5. status.mem >= budget is now its own branch with its own words ("The memory budget of qex on this machine is 1GB, so qex has no larger claim to give"), and a degenerate growth has a third branch naming growth. Both texts are asserted now, not just the variant.
• 6. qex rerun prefers status.mem/status.cpu when larger than the spec, and carries claim_source. Asserted in the e2e.
• 7. unexplained_kill_note only fills status.error when it is empty.

Tests

New: the_queue_tests_the_claim_that_the_job_holds_now (unit), a_multiplier_that_gives_the_same_claim_stops_the_ladder (unit), the extended the_out_of_memory_record_is_read_back and the_mark_of_a_kill_by_a_command_can_be_cleared (unit), plus e2e a_kill_for_memory_with_no_limit_reports_but_does_not_run_the_job_again, a_raised_claim_waits_for_capacity_in_the_queue and a_mark_from_one_attempt_does_not_decide_the_next_attempt.

The simulated-kill helper's doc comment now also records that the planted evidence says job, i.e. it stands in for [enforce] mode being on, and why these tests must not use that mode (the systemd re-exec would lose the test's XDG_* values and write to the user's real state directory).

cargo fmt --all                            clean
cargo clippy --all-targets -- -D warnings  clean
cargo test --bins                          187 passed, 0 failed
cargo test --test e2e -- --test-threads=2   77 passed, 0 failed (64s)