feat: the group id of a pipeline names every stage - #11
Open
stephenc wants to merge 3 commits into
Open
Conversation
`qex pipeline` writes a group id to stdout, which is the obvious value to
keep. Every command except `qex list --group` then answered "there is no
job with the id ..." with the code 127. 127 is also the code for a value
that names nothing, so a script could not separate the two, and the
documented way to use a pipeline ended with the user finding the last
stage by hand.
The resolver now reads a group as well as a job. A group id, the start of
a group id, and the name of the pipeline each give EVERY stage, in the
order of submission:
GROUP=$(qex pipeline ci.toml)
qex wait $GROUP # wait for every stage
qex status $GROUP # the state of every stage
qex kill $GROUP # stop every stage
qex clean $GROUP # delete every record
`--needs $GROUP` waits for the whole pipeline in the same way.
Three rules keep this safe:
- A job comes first. A word that names a job AND a pipeline gives an
error that shows both readings, because qex must not choose one and
kill the wrong work.
- `qex status --json` gives an array for a pipeline and one object for
one job. A script that reads one job does not change.
- `qex logs` reads one job, so it refuses a pipeline and names the
stages. qex must not choose a stage for the reader.
`qex wait` with no coordinator keeps its earlier behaviour: it reads the
state directory, which holds no group, so a group needs a coordinator.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KNvagiMEU3myn8EXGaGEM9
An adversarial review of this branch found three faults, and the first is the dangerous one. **A group name is not unique.** A pipeline takes its name from its file, so `qex pipeline ci.toml` twice gives two runs that the word `ci` both names. The resolver gave every match, so `qex kill ci` stopped the work of two runs and the user named one. `qex clean ci` deleted both. A short group id had the same fault, because two ids can start with the same characters. The resolver now refuses a word that gives more than one run, and it shows the group id and the stage count of each. This is the same rule that the resolver already had for a job name. **`qex kill $GROUP` gave the code 1 in the ordinary case.** The stages of a pipeline stop in order, so at the moment a user stops a pipeline the early stages have usually finished, and each one gave a fault. The command succeeded only while every stage still operated. A stage that already stopped is now information when the user named the WHOLE pipeline, and it is still a fault when the user named that one job. **`qex wait` hid the true message.** It kept a value that the resolver refused and let the wait fail later, so a word that named a job AND a pipeline came back as "there is no job with the id x". The user read that the value named nothing, and it named two things. Three more corrections from the same review: - `qex status --json` chose an array or an object from the NUMBER of jobs. A pipeline of one stage gave an object, against the documentation, and a script that reads `.[0]` of a group would break on the day a pipeline has one stage. The shape now comes from what the user named. - `--needs $NAME` for a pipeline tested each stage, so it refused as soon as one stage stopped, which is the ordinary case. The test now applies to the pipeline as one unit: it refuses when EVERY stage stopped, which is what "a run of an earlier day" means. - `qex clean` gave the code 1 where every other command gives 127 for the same fault, and its message called a pipeline a job. The unit test for the order of the stages proved nothing: the fixture was already in the order that it asserted, and the test passed with the sort deleted. The fixture now holds the stages in a different order. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KNvagiMEU3myn8EXGaGEM9
A second review found that the earlier correction went too far. It made every "wrong state" answer information when the user named a pipeline, and two of those answers are real faults: - `qex kill $GROUP` reported success for a stage that WAITS in the queue, and left it there. The queue then started that stage after the command said that it stopped every stage. A script that reads the code 0 and cleans up thus removed the files of work that was about to start. - `qex cancel $GROUP` reported success for a stage that OPERATES, and cancelled nothing. Both now keep their code. Only a stage that ALREADY STOPPED is information, because the stages of a pipeline stop in order and the early stages have usually finished. `qex kill $GROUP` also takes a stage that waits out of the queue, because the documentation says that the command stops every stage. `qex kill $ID` for one job that waits still gives the fault and names `qex cancel`: that user asked about that one job. Two smaller corrections from the same review: - The remedy for `--needs` on a pipeline that stopped wrote `qex pipeline <name>.toml`. The name comes from `--name` or from the file, and a pipeline file can be YAML or JSON, so that file frequently does not exist. A remedy that names a file that is not there is not a remedy. - `qex list --group` had no test for a word that names two runs. It shows them all, because it reads and deletes nothing, but a reader who does not know that saw one pipeline that ran each stage two times. It now says how many runs the word names, and it gives the group id of each. The documentation now also says that a group needs a coordinator: the records on the disk hold no group, so `qex wait $GROUP` after the coordinator retires says that there is no job with that id. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KNvagiMEU3myn8EXGaGEM9
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes the first item of #10.
qex pipelinewrites a group id to stdout — the obvious thing to keep — and thenqex wait $G,qex status $G,qex kill $Gandqex clean $Gall answeredthere is no job with the id ...with the code 127. Onlyqex list --grouptook it. 127 is also the code for a value that names nothing, so a script could not tell the two apart, and the documented way to use a pipeline ended with the user finding the last stage by hand.The resolver now reads a group as well as a job. A group id, the start of a group id, and the name of the pipeline each give every stage, in the order of submission:
--needs $GROUPwaits for the whole pipeline in the same way.Three rules keep it safe
qex status --jsongives an array for a pipeline and one object for one job. A script that reads one job does not change.qex logsstill reads one job. It refuses a pipeline and names the stages, because qex must not choose a stage for the reader.qex waitwith no coordinator keeps its earlier behaviour: it reads the state directory, which holds no group, so a group needs a coordinator.Tested
176 unit tests (4 new, covering the order of the stages, the short id, the name, the both-readings error, and the empty value that must not select everything). 73 e2e tests, 2 new: one that takes a pipeline through wait / status / clean, and one that kills a group of two running stages and confirms both reach
killed.cargo fmt,cargo clippy --all-targets -D warnings,cargo test --bins,cargo test --test e2e -- --test-threads=2all green locally. GitHub Actions was in a major outage when this was pushed, so CI has not run yet.