feat: run a command from the config file when a job stops

[stephenc]

Closes part of #10 ("Smaller, agreed": A hook when a job stops, in the
configuration and not on the job).

What it does

[hooks] on_stop names a command that qex runs when a job reaches its final
state. A person who started a job of four hours and walked away learns that it
stopped.

[hooks]
on_stop = ["notify-send", "a qex job stopped"]
on_stop_states = ["completed", "failed", "killed", "timeout", "oom"]
timeout = "30s"

With no [hooks] section there is no hook, and nothing changes.

The decisions, and the reason for each

Which process runs the hook: the supervisor, at the end. The supervisor
exists for each job that ran, and it knows the result first. The coordinator
can stop and start again while a job runs, so a coordinator that ran the hook
would miss every job of that period. The coordinator runs the hook for the jobs
that no supervisor sees: cancelled, skipped, a job whose supervisor left no
result, and a job that a restart found dead. It does that in a thread of its
own, because it makes those jobs terminal while it holds the lock of the queue.

One run for each job: a claim file, and not a code path. There is no single
place in the program that every terminal transition passes through. hook.ran
gives the guarantee instead: each process makes that file with create_new
before it starts the hook, and the operating system gives the file to one
process only. The file lives beside the record of the job, so the guarantee
holds after a restart of the coordinator as well. A notification that arrives
two times teaches a person to ignore every notification.

What the hook receives: the environment. QEX_JOB_ID, QEX_JOB_NAME,
QEX_STATE, QEX_EXIT_CODE, QEX_SIGNAL, QEX_ELAPSED_SECS, QEX_CWD,
QEX_JOB_DIR, QEX_ATTEMPTS, QEX_MAX_RSS and QEX_TAGS. The set answers the
questions that a person asks when the message arrives: which job, what
happened, how long, and where do I look now. A hook that needs more reads
spec.json and status.json in QEX_JOB_DIR, so the list stays short. A
variable with no value is empty text and not an absent variable, so a shell line
needs no test.

An argument list, and not a shell line. The rest of qex takes an argument
list and starts no shell, and qex help says so. The hook keeps that position,
and it uses the same escape that the job file documents: name the shell,
["bash", "-lc", "..."]. The exception is not necessary, because the values of
the job arrive as variables. The shell that the user names expands
$QEX_JOB_NAME from the environment; qex never builds that text.

Security. The command is in the user's own config file, so it is the user's
command. The data of the job is NOT the user's: a job name comes from whoever
submitted the job. Every value therefore goes in the environment, and qex builds
no command line from any of it. A job named ; rm -rf ~ gives a hook a variable
with those letters in it, and never a command. A unit test holds that.

Which jobs fire it. on_stop_states selects them. The default list holds
each state of a job that ran: completed, failed, killed, timeout, oom.
cancelled and skipped need a word from the user, because the person
cancelled the job, and one failure in a pipeline of twenty stages would give
twenty messages. A job that failed and ran again gives ONE message, with the
final result: a retry returns the job to queued, which is not a final state.
An unknown state name, or a state that is not final, gives an error at start.

A hook cannot damage the queue. qex starts it after the final state is on
the disk. The job thus has its result, its claim has left the budget, and the
next job starts before the hook does anything. The hook runs in a process group
of its own; at [hooks] timeout qex sends TERM to that group and KILL two
seconds later, so a hook that starts children leaves nothing behind. The output
goes to hook.log in the job directory with mode 0600, and the time limit
bounds how much a hook can write. A hook that fails, or that does not exist,
writes a line in the log and changes no job.

What I measured

• A hook that succeeds: it started 52 ms before qex wait gave its answer
  to the shell. The hook is not in the path of the result.
• A hook that hangs (sleep 300, limit 2s) on the first job, with a budget of
  one core: the second job went from submit to a result in 0.53 s. The
  hanging hook delayed the next job by nothing, and the first job still says
  completed.
• With no [hooks] section, the added work is one test of an empty list.

What I tested

New unit tests in src/hook.rs: one run for each job with three calls; the
variables that the hook receives; a job name with shell characters that must not
become a command; a hook that hangs and stops at its limit; a hook that does not
exist; the filter; a job that did not stop. New unit tests in src/config.rs:
no hook by default, the default filter, and a state name that qex refuses.

New end-to-end tests in tests/e2e.rs:

• a_job_that_stops_runs_the_stop_hook_one_time_with_its_result
• a_stop_hook_that_hangs_holds_neither_the_job_nor_the_queue
• the_configured_states_select_the_jobs_that_run_the_stop_hook (this one goes
  through the coordinator, for a skipped job)

cargo fmt --all                             clean
cargo clippy --all-targets -- -D warnings   clean
cargo test --bins                           182 passed, 0 failed
cargo test --test e2e -- --test-threads=2    74 passed, 0 failed

Notes

• The version in Cargo.toml moves to 0.8.0.
• Documentation: docs/reference.md, docs/design.md (the files of a job),
  docs/security.md (the mode of hook.log, and the reason for the
  environment), and qex help config. qex config show now names the hook,
  because a command that starts on each job and that the reader forgot is a
  program with no visible cause.
• There is no JSON Schema for the config file in the repository. The schemas
  cover the job file, the status and the pipeline file, and none of them holds a
  hook field, because the hook is not a property of a job.
• The config parser refuses an unknown key, so a config file with [hooks]
  stops a coordinator of 0.7.1 with a parse error. That is the behaviour of
  every new section, and qex compares the versions and warns.

🤖 Generated with Claude Code

https://claude.ai/code/session_01KNvagiMEU3myn8EXGaGEM9

[stephenc]

Review answered in 889d19b. Version moved to 0.8.1, so this build reports a
different number from the first one.

BLOCKING 1 — a NUL in a job name. Fixed in variables(): each control
character becomes a space, so the hook always starts. The reviewer's
reproduction now gives a b|completed in the hook, not a lost message. The
escape sequence case goes the same way, so a job name cannot send an escape to a
screen. Regression test: a_job_name_with_a_control_byte_still_runs_the_hook.
The remedy in the message also names the program from on_stop now, instead of
sending the reader to a file that is correct.

2 — at most once, and the documentation. Took the free fix: the terminal arm
of reap now calls fire_detached. The window is the microseconds between the
terminal write and fire, so I could not build a deterministic test for it; the
new e2e test covers the reachable half of the same path (a supervisor that a
signal stops, where the coordinator makes the job failed and runs the hook)
and proves it still runs exactly one time. The documentation now states the true
guarantee in docs/reference.md, qex help config, docs/design.md and the
module comment: qex never runs the hook two times, and it loses one message when
a process stops between the record of the run and the run. The order is
deliberate and the comment says why.

3 — the list of states. skipped added to the message in src/job.rs, and
[hooks] on_stop_states no longer defers to that general list: it names the
seven final states only, so no message offers a value that the next line
refuses.

4 — the verdict reaches no command. The verdict now goes into hook.log as
a qex: line, and qex logs <id> --hook reads that file. A job that ran no hook
gets a sentence, not silence. Tests:
the_verdict_of_qex_goes_into_the_log_of_the_hook, and the e2e test asserts the
line through the CLI.

5 — the output had no limit. It has one now: 1MB. qex stops the hook when
the file goes above it and cuts the file. Measured with the reviewer's case
(yes, default 30s timeout): hook.log is 1,048,831 bytes and the state
directory is 1.1M, in place of 3.7GB at three seconds. Test:
a_hook_that_writes_without_a_stop_is_stopped_and_its_log_is_cut.

6 — the coordinator path loses its limit. The comment says so now, in
capitals, with the reason and the list of the states that use that path.

The deny_unknown_fields mitigation was contained, so I did it: the parse
error now says "A newer qex can have added a section or a key that this qex does
not know. Install the newer qex, or delete the lines that the fault below
names". One string in Config::load, and it covers every future section.

The cgroup. The spawn-failure path now leaves and removes the cgroup of the
job before the hook runs, so the hook never inherits the job's memory.max.
That also closes the pre-existing leak on that path.

cargo fmt --all                             clean
cargo clippy --all-targets -- -D warnings   clean
cargo test --bins                           185 passed, 0 failed
cargo test --test e2e -- --test-threads=2    75 passed, 0 failed

[stephenc]

Second review answered in c3ceeca. Version moved to 0.8.2.

BLOCKING — a stale configuration ran a command that the user deleted. Took
the contained route. hook::fire now reads the config file itself, at the
moment that the job stops, and each caller gives the job only: fire(dir, status) and fire_detached(dir, status). state.cfg is no longer used for the
hook anywhere, and the supervisor no longer carries the copy that it read hours
before at the start of a long job. The comment on fire records the reason in
capitals: a stale configuration made qex do nothing before this feature, and
here it made qex EXECUTE.

Measured with the reviewer's two directions, one coordinator, no restart:

• Hook deleted while the coordinator operates: the next job wrote no line, and
  its directory holds no hook.ran and no hook.log. Before: OLD-HOOK-second.
• Hook added while the coordinator operates, on_stop_states = ["skipped"]: the
  skipped dependent wrote NEW-skipped. Before: nothing.

Regression test: a_stop_hook_that_the_user_deletes_does_not_run_again covers
both directions in one test, and the second half goes through the coordinator
path (a skipped job has no supervisor). docs/reference.md and qex help config now state the behaviour: qex reads the file at each job that stops, and
you do not restart the coordinator.

The 1MB cap for a fast hook. qex now tests the size again after the loop,
whatever ended it, and cuts the file there. Measured with head -c 3000000 /dev/zero, which stops inside one interval: hook.log is 1,048,576 bytes and
the verdict says "wrote more than 1MB before it stopped". A hook that qex stops
for time and that also wrote too much gets both facts in one line. Test:
a_hook_that_writes_a_large_file_quickly_also_meets_the_size_limit. The three
places that state the cap now also say that qex cuts the log of each hook.

killpg after the child was reaped. stop() no longer waits during the
grace time. It sleeps, then sends KILL to the group, then waits. The first
process thus stays in the process table until after the last signal, so the
machine cannot give its number to different work. That is the same rule as
wait_without_reaping in the supervisor, and the comment names it. The cost is
the full grace time on the path of a hook that qex must stop.

The verdict in a cut log. note() writes a newline before the text, so the
verdict is a line of its own. qex logs --hook --tail 1 now gives the verdict
and not a megabyte. The new size test asserts that the last line starts with
qex: .

--hook --json and --hook --stdout. With --json, the sentence goes to
stderr and the document still goes to stdout: {"hook": "", "id": "..."}. A
reader that asks for JSON always receives JSON. --hook now conflicts with
--stdout and --stderr as well as --follow, so an option that has no
meaning gives an error and is not ignored in silence.

PR #25. Understood: at the rebase or the merge I take the main version of
the config parse error and drop mine. I did not touch it in this commit.

cargo fmt --all                             clean
cargo clippy --all-targets -- -D warnings   clean
cargo test --bins                           186 passed, 0 failed
cargo test --test e2e -- --test-threads=2    76 passed, 0 failed (39.3s)