feat(cli)!: validate grades your change by default - #94
Merged
Conversation
validate's verdict covers the resulting branch, not your change. On a branch that already carries findings — the normal state of a large imported graph — that makes 'validate && commit' unusable however correct the change is, and gives a caller no way to tell 'I broke this' from 'it was already like this'. Measured on a real branch: an empty stage returns 99 findings and exit 5. With the flag, the same empty stage exits 0, and a stage that introduces one unwired input exits 5 naming only that one, with the 2 inherited findings listed separately as not caused by the change. The split comes from two server answers — the branch as it stands, and the branch with the stage applied. The baseline is prepare() over an empty stage, which is pure and already a supported request shape, so the real stage is never touched, swapped or restored. Fail-closed, because the two error directions are not symmetric. An inherited finding misread as introduced is loud: a spurious non-zero exit someone complains about the same day. An introduced finding misread as inherited is silent: it still prints, the verdict reads clean, and the gate passes where today it stops. So anything ambiguous is introduced. Identity is (code, raw locator), keyed BEFORE path resolution — resolution is presentation, it can be partial, and a stale index can fold two ids onto one path. Counted as a multiset rather than a set: a locator legitimately carries more than one finding, so with a set a locator going from one finding to two would register as no change, hiding the new one in the silent direction. The two answers must describe the same branch version. If the branch moved between the reads they describe different graphs, and attributing across them would blame the stage for someone else's commit or hide one of its own behind theirs — so it refuses to partition and says why. A conservation check on the buckets does the same for a bug in this code. JSON carries the three buckets as separate arrays rather than one list with a flag, and valid is redefined to match the exit code, with the server's whole-branch answer kept under its own key — otherwise the most gateable field in the payload would disagree with the exit status beside it. Default behaviour is unchanged; this ships behind the flag. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The verdict covered the resulting branch, so on a branch that already carried findings the advertised 'validate && commit' gate could never fire. Measured: an empty stage on a real imported graph returned 99 findings and exit 5. Bare validate now reports and gates on the findings your staged change adds. --whole-branch grades the whole graph and follows the server's verdict verbatim, which is the old behaviour kept intact for the branch-health question. BREAKING: exit 5 changes meaning for existing callers. It was a pure function of the server's valid field; it now follows the introduced set. A caller running bare validate as a CI branch-health check gets 0 where it got 5, for the same command against the same branch — so the change leaves a runtime trace on stderr in both output modes whenever there are inherited findings and none of them are yours. A release note is not a runtime signal. Six texts asserted the old rule and go stale together: the module doc, the error-findings doc, the VALIDATION exit constant, the validate help text, the guide, and the contributor guide's exit bullet. All amended here rather than left for someone to find later. The narrowed rule they now state: the server owns each verdict, and this client owns only which verdict it gates on — every finding is still the server's, and no rule is re-implemented. The guide test that pinned the old whole-branch semantics failed on this change, which is what it was for; it now pins the new scoping and asserts the manual baseline workflow it replaced is gone. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…othing A review mutated three of these tests and found them decoration. Dropping 'code' from the finding identity entirely passed all 421 tests. The test named for it used base=[UNSAT@a], staged=[UNSAT@a, MISMATCH@a], where multiset counting hands the second entry to introduced whichever key is used. The discriminating case swaps the code between the reports: with the real key that is one resolved and one introduced; keyed on the locator alone it collapses to a single INHERITED finding, which is the silent direction. Now tested, and the mutation fails it. The multiset was only half covered: replacing the baseline tally with a plain insert survived every test, because no test used a baseline carrying two findings on one key. Now it does. The conservation checks were tautologies — the loops push each finding exactly once, so comparing bucket lengths to input lengths is true by construction, and a review rewrote them to '> total + 9999' with nothing failing. They now compare independently re-derived tallies, key by key, and are exercised directly with dropped, invented and mis-attributed buckets. What they deliberately do NOT catch is a finding moved between introduced and inherited: those are summed together against the staged report, and attribution is the bucket tests' job. The version guard printed a warning and fell through to whole-branch semantics, where the plan called for a retry then a conflict. It now retries once — a single concurrent commit is the common case — and returns a version conflict when the branch is still moving, which is what that condition means everywhere else here and is retryable. The partitioned JSON carried a 'located' array whose finding_index pointed into a 'findings' list the payload does not contain, so an authored path could not be joined to a bucketed finding at all. Each bucket now carries its own path and path_complete inline. And validate no longer issues a second request when the stage is empty: an empty changeset cannot introduce anything, so the baseline is the staged report by construction. That matters because validate is write-scope gated and rate limited per principal, and it was the one case where the extra call bought nothing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A review deleted the stderr note, and separately forced whole-branch on, and the whole suite passed both times. Neither the signal nor the default was pinned by anything — the one test that mentioned the note asserted on stdout, not on the stderr line it was named for. An integration test now runs the real binary against a server whose baseline and staged reports are identical, which is exactly the 5-to-0 transition the note exists to announce, and asserts both the exit code and the note. Both mutations fail it. The note also fired on too narrow a condition. It now fires whenever the answer CHANGED — comparing the server's verdict against the computed one — because two transitions escaped the old check: an introduced set holding only warning-severity findings, and a valid:false report with no findings to bucket. Neither is reachable against today's server, but the warning arm is kept in the contract deliberately, so the guard must not assume it away. Also: run()'s doc still described the old exit contract; the inherited-count line now names --whole-branch as the way to list them, since the detail only exists there; and disagreement_warning says which response it checks and why the baseline is not cross-checked. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The partition landed on main, so this branch keeps only the flip. cli.rs is rebuilt from main (which carries the stage verb group) plus the flag change, and the integration tests are main's set plus this branch's own. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on the partition PR — merge that one first; this deletes the flag it adds.
The verdict covered the resulting branch, so on a branch that already carried findings the advertised
validate && commitgate could never fire. Measured: an empty stage on a real imported graph returned 99 findings and exit 5.Breaking: exit
5changes meaningIt was a pure function of the server's
valid; it now follows the introduced set. A caller running barevalidateas a CI branch-health check gets0where it got5, for the same command against the same branch.That's why the note above goes to stderr in both modes whenever there are inherited findings and none are yours. A release note is not a runtime signal, and a CI log is where someone will actually look.
Wants a minor version bump and a release note leading with this.
Six texts, not two
An earlier draft of the plan said "both texts must be amended", which an implementer would have followed literally. Six assert the old rule and go stale together — the module doc, the error-findings doc, the
VALIDATIONconstant, thevalidatehelp text,hydrate guide, and the contributor guide's exit bullet. All amended here.The narrowed rule they now state: the server owns each verdict; this client owns only which verdict it gates on. Every finding is still the server's, and no rule is re-implemented —
--whole-branchstill reportsresponse.validverbatim, pinned by a test.A test earned its keep
guide_says_the_validate_verdict_covers_the_whole_branchfailed on this change — which is exactly what it was written for. It now pins the new scoping and asserts the manual baseline workflow it replaced is gone from the guide.421 tests.
fmt,clippy -D warningsclean. Both paths verified against the live service.