feat: make a submission idempotent with --dedupe-key

[stephenc]

Closes part of #10: nothing makes a submission idempotent.

An agent that loses its context runs its script again. Without a key, qex starts
a second copy of a four-hour training run beside the first copy. The agent page
told that agent to run qex list and look — a proxy check, which is the fault
this tool exists to remove.

ID=$(qex submit --dedupe-key train:$(pwd) -- uv run train.py)
qex wait $ID

The second run of that script gives the same id and exits with the code 0.

What I decided, and why

"Already operates" means a job that has not stopped. A key holds a job while
that job waits in the queue or operates. When the job stops, the key is free and
the next submission starts a new job. A key that held a job for ever would give
an agent the id of a job of yesterday, and that answer looks like a success. The
rule is one sentence: the key stops a second copy of the work, and it does
nothing else.

The answer is selectable: --dedupe-window 1h. The window keeps the key of a
job that succeeded for that time, for a caller that wants a completed result
to count. A job that did not succeed never keeps its key, whatever the window
is. A window that blocked a retry would make the option dangerous: the one remedy
for a failure is another run. The window comes from the submission that asks, and
not from the job that holds the key, so the coordinator keeps no policy of its
own and the caller says how old an answer it accepts.

The key is also free when the record goes. qex clean and qex gc delete the
key with the record, because a key that named a job with no record would give an
id that qex status cannot answer.

How a caller tells the two apart. stdout holds the id alone and the exit code
is 0 in both cases, because ID=$(qex submit ...) must not change. The
difference is on the other stream:

qex: this submission started no job. The dedupe key `train:/home/me/p` gives
the job 7f3c8a12-..., and that job is in the state `running`.

For a script that must know whether it started the work, qex submit --json
writes {"id": "...", "deduplicated": true}. qex run refuses --json with a
message, because its stdout holds the output of the job. qex status <id> shows
the key of a job, so a caller can see which key gave it an id.

The race. The check and the reservation are ONE lock operation in the
coordinator (State::dedupe, a map from key to job id, read and written inside
handle_submit while the state lock is held). The reservation goes in before the
lock is given back, so a second submission in the same moment finds the id. A
reservation that names a job with no record yet counts as taken; that case
cannot mean "the record went away", because clean deletes the entry with the
record. A submission that qex then refuses, or whose record it cannot write,
frees the key again.

Where the key lives. In JobSpec and in JobStatus, so qex list --json and
qex status show it and it survives a coordinator that stops. recover() gives
each key back to its job at start: a job that has not stopped wins over a job
that has, and after that the latest job wins.

The capability. dedupe is in capabilities::ALL and in required_by. This
is the most dangerous option to ignore in silence: an earlier coordinator would
start the second copy, give a new id, and say nothing.

qex rerun clears the key. That command exists to run the work again; with
the key it would give the first job's id and do nothing.

A pipeline stage has no key. A key on one stage would answer for that stage
alone, and the stages after it would wait for a job of an earlier run — two runs
mixed in one graph. A pipeline needs one key for the whole group, which is a
different feature. The Stage type holds no such field, so a file that names one
gives an error and qex accepts no part of it in silence. dedupe_key and
dedupe_window are job-file fields, because qex submit --job train.toml is
one of the commands that a script repeats.

What I measured

• The full e2e suite: 76 tests pass in 20.51s with --test-threads=2.
• The unit tests: 181 pass.
• 20 submissions with one key started at the same moment: one job, one job
  directory, and 20 equal ids.

What I tested

New e2e tests in tests/e2e.rs:

• a_second_submission_with_one_key_gives_the_first_job_and_starts_no_job — the
  same id, the code 0, one job in qex list, one job directory, the reason on
  stderr, and the key in the record.
• many_submissions_with_one_key_at_once_make_one_job — 20 submissions at the
  same moment give one job and one id.
• a_job_that_stopped_frees_its_key_and_a_window_keeps_it — a job that stopped
  frees the key; the window keeps the key of a job that succeeded; a job that
  failed frees its key inside the window.
• submit_json_says_if_this_command_started_the_work.
• a_key_stays_with_its_job_after_the_coordinator_stops — the coordinator is
  killed (its pid comes from qex info --json, never from a search of the
  process list) and the new coordinator gives the key back to its job.

New unit tests: the rules of State::dedupe_holder for each job state, the
reservation that has no record yet, the release with the record, the capability
refusal (a_dedupe_key_is_refused_by_a_coordinator_that_has_no_dedupe), and the
resolution of the key and the window from the command line and the job file.

Also updated: docs/agents.md (a section that tells an agent to use the key and
NOT to read qex list and decide), docs/reference.md, README.md,
skills/qex/SKILL.md, qex help agents, qex help job-file, the job schema and
the status schema. The repository generates no shell completions on main.
Cargo.toml moves to 0.8.0.

🤖 Generated with Claude Code

https://claude.ai/code/session_01KNvagiMEU3myn8EXGaGEM9

[stephenc]

Review findings addressed in 50c7631.

1. qex run on the dedupe path no longer stops a job it did not start. I chose the refusal over the warning: a caller cannot know the id belongs to somebody else, and stopping a four-hour run of another agent on a Ctrl-C is the exact harm qex exists to prevent. stream_until_done takes owns_job, which is !deduplicated from the wire field that qex run previously discarded. When it is false, a signal stops the wait, prints the job id and qex kill <id>, and gives the code 124 — the code qex already documents as "your wait stopped, and the job continues". qex run also says the rule when it attaches, and it now prints that line after the handler is installed, so the message is proof that the rule is active and not the default disposition.

New e2e tests:

• a_signal_to_a_deduplicated_run_stops_the_wait_and_not_the_job — reproduces the reviewer's measurement: agent A's job stays running, B exits 124, and B's stderr names qex kill <A's id>. It waits for the attach message before it signals, so there is no window in which the signal meets the default behaviour.
• a_signal_to_a_run_that_started_its_job_stops_the_job — the owner path keeps its earlier behaviour. There was no test for it before, so a later change could have removed it in silence.

Documented in docs/reference.md (new qex run with a key subsection), docs/agents.md, qex help agents (including the general "Ctrl-C stops the job" line, which now names the exception), and skills/qex/SKILL.md.

2. Whose window applies is now stated. "The window of the submission that asks applies, and not the window of the job that holds the key" — with the reviewer's three-command bypass as the worked example, and the reason: the window is a question (how old an answer do I accept?), not a property of the earlier job, which keeps policy out of the coordinator. Each place says it concerns a job that already succeeded only, so the no-double-start property is untouched, and each says to give the same window in every command that shares a key. In docs/reference.md, docs/agents.md, qex help agents, skills/qex/SKILL.md and the JobSpec::dedupe_window doc comment. The table row now reads "succeeded inside the --dedupe-window of the new submission".

3. The comment now points at the right field. JobStatus::dedupe_key says it is for a reader and the schema, that recovery reads spec.json, and that JobSpec::dedupe_key must not be deleted. JobSpec::dedupe_key says it is the field recovery reads, and the comment in recover() names spec.json.

Test results after the changes:

• cargo fmt --all --check clean
• cargo clippy --all-targets -- -D warnings clean
• cargo test --bins — 181 passed, 0 failed
• cargo test --test e2e -- --test-threads=2 — 78 passed, 0 failed, 47.77s

Understood on the merge order: this must go in after #22 (fix/retry-latch).