Phase 1: the exit-code contract and machine-readable output (ADR-0007) - #24
Merged
Conversation
Three findings, all verified rather than argued. The verdict range moves from 70-73 to 80-83. The design justified 70-73 as "clear of sysexits' common values"; sysexits.h defines EX_SOFTWARE 70, EX_OSERR 71, EX_OSFILE 72 and EX_CANTCREAT 73, four consecutive values inside its 64-78 block. The collision at 70 is the damaging one: the convention reserves it for an internal error in the program itself, which is what this program means by 82. A wrapper dying of its own fault at 70 would have been reported as a regression that never happened — the manufactured verdict forbidden everywhere else, arriving through the exit code rather than through verdict logic. Nothing had been published, so the correction was free. Precedence is now specified: 82, then 80, then 81, then 83, then 0. Simultaneity is routine — a suite can carry a failing exam, a rotted pin and an errored one at once — and without an order two conforming implementations disagree, with a real regression resolving to the stale code and being waved through as the worst case. Trailing arguments were silently ignored, so `assayer version --jsonn` printed human text and exited 0: a script asking for a machine-readable document got prose and a success code. Flags are now checked against what each command accepts. Machine-readable output starts with `version --json`. Documents carry the schema identifier, a revision that moves whenever an emitted shape changes, the stability tier, and the list of schemas the build can produce, so a consumer can discover what it is talking to without running anything that costs money. Schemas are generated from the types that produce the documents and asserted against the committed files, rather than hand-maintained beside them and checked by a validator this module would have needed its first dependency for. Identifiers are rooted at this repository. An earlier draft used assayer.dev on the assumption that nothing was served there; that was not checked and is false — the domain is registered and serving an unrelated product of the same name. schemas/ joins the architecture guard's scan roots: a committed schema enumerating adapter names would otherwise name harnesses in a public artifact that nothing was looking at. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ew64uotJqtvVDN3twMAVEQ
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Third of three Phase 1 records. Two independent adversarial reviews found three
things worth catching before anything was published.
The verdict exit codes were wrong, in the one direction that matters
The design justified 70–73 as "clear of sysexits' common values". Verified
against
sysexits.hon this machine:Not near sysexits — four consecutive values inside its 64–78 block. And the
collision at 70 is semantically inverted: the convention reserves it for an
internal error in the program itself, which is what Assayer means by 82, not 80.
A wrapper dying of its own fault at 70 would have been reported as a regression
that never happened — the manufactured verdict this design forbids everywhere
else, arriving through the exit code rather than through any verdict logic.
Moved to 80–83, above the sysexits block and below the shell's 126/127.
Nothing had been published, so this was free today and a breaking change to
strangers' scripts after the first release.
Precedence was unspecified
A suite can carry a failing exam, a rotted pin, and an errored one at once — the
decision rule makes that routine. With no stated order, two conforming
implementations disagree, and the worst case is a real regression resolving to
the stale-pin code and being waved through. Now specified and tested:
82 > 80 > 81 > 83 > 0, with 82 first because it says the other answers could
not be trusted.
A shipped bug: trailing arguments were ignored
A script asking for a machine-readable document got prose and a success code.
Flags are now checked against what each command accepts; that case exits 2.
Machine-readable output starts
assayer version --jsonemits the first document. Each carries its schemaidentifier, a
revisionthat moves whenever an emitted shape changes, thestability tier, and the list of schemas the build can produce — discovery
without running anything that costs money.
Schemas are generated from the types, not hand-maintained beside them, and
asserted against the committed files. A hand-written schema plus an external
validator is two descriptions that can be wrong together: rename a field,
regenerate the sample, and it still validates. This is the arrangement the repo
already uses for its linter rules, and it needs no dependency —
go.modstaysempty.
Verified by planting a field: both the schema check and the golden fail.
assayer.devis someone else's productThe draft rooted schema identifiers there, on the stated assumption that nothing
was served at that address. Not checked, and false — it is registered and
serving "Assayer – Production Readiness Review for Vibe-Coded Apps". Identifiers
are now rooted at this repository. The namesake collision is worth a separate
look at ADR-0002; this PR only stops the contract depending on it.
Stated rather than implied
The event stream and verdict objects are not built. They land before Phase 1
closes, with the components that produce them — and no command returns a verdict
code until a machine-readable verdict object ships alongside it, because an alarm
with no readout leaves scraping human output as the only option, and those
scrapers become an unversioned contract.
The exam format's schema is also deferred, carrying a real question: the format
is TOML, the stdlib cannot parse TOML, so validating it needs either this
module's first dependency or a different format. Recorded so it stops being
invisible.
make verifygreen: coverage 90.2%, 0 lint issues, 0 vulnerabilities.