feat(cli): validate --introduced reports only what your stage caused - #93
Merged
Conversation
validate's verdict covers the resulting branch, not your change. On a branch that already carries findings — the normal state of a large imported graph — that makes 'validate && commit' unusable however correct the change is, and gives a caller no way to tell 'I broke this' from 'it was already like this'. Measured on a real branch: an empty stage returns 99 findings and exit 5. With the flag, the same empty stage exits 0, and a stage that introduces one unwired input exits 5 naming only that one, with the 2 inherited findings listed separately as not caused by the change. The split comes from two server answers — the branch as it stands, and the branch with the stage applied. The baseline is prepare() over an empty stage, which is pure and already a supported request shape, so the real stage is never touched, swapped or restored. Fail-closed, because the two error directions are not symmetric. An inherited finding misread as introduced is loud: a spurious non-zero exit someone complains about the same day. An introduced finding misread as inherited is silent: it still prints, the verdict reads clean, and the gate passes where today it stops. So anything ambiguous is introduced. Identity is (code, raw locator), keyed BEFORE path resolution — resolution is presentation, it can be partial, and a stale index can fold two ids onto one path. Counted as a multiset rather than a set: a locator legitimately carries more than one finding, so with a set a locator going from one finding to two would register as no change, hiding the new one in the silent direction. The two answers must describe the same branch version. If the branch moved between the reads they describe different graphs, and attributing across them would blame the stage for someone else's commit or hide one of its own behind theirs — so it refuses to partition and says why. A conservation check on the buckets does the same for a bug in this code. JSON carries the three buckets as separate arrays rather than one list with a flag, and valid is redefined to match the exit code, with the server's whole-branch answer kept under its own key — otherwise the most gateable field in the payload would disagree with the exit status beside it. Default behaviour is unchanged; this ships behind the flag. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…othing A review mutated three of these tests and found them decoration. Dropping 'code' from the finding identity entirely passed all 421 tests. The test named for it used base=[UNSAT@a], staged=[UNSAT@a, MISMATCH@a], where multiset counting hands the second entry to introduced whichever key is used. The discriminating case swaps the code between the reports: with the real key that is one resolved and one introduced; keyed on the locator alone it collapses to a single INHERITED finding, which is the silent direction. Now tested, and the mutation fails it. The multiset was only half covered: replacing the baseline tally with a plain insert survived every test, because no test used a baseline carrying two findings on one key. Now it does. The conservation checks were tautologies — the loops push each finding exactly once, so comparing bucket lengths to input lengths is true by construction, and a review rewrote them to '> total + 9999' with nothing failing. They now compare independently re-derived tallies, key by key, and are exercised directly with dropped, invented and mis-attributed buckets. What they deliberately do NOT catch is a finding moved between introduced and inherited: those are summed together against the staged report, and attribution is the bucket tests' job. The version guard printed a warning and fell through to whole-branch semantics, where the plan called for a retry then a conflict. It now retries once — a single concurrent commit is the common case — and returns a version conflict when the branch is still moving, which is what that condition means everywhere else here and is retryable. The partitioned JSON carried a 'located' array whose finding_index pointed into a 'findings' list the payload does not contain, so an authored path could not be joined to a bucketed finding at all. Each bucket now carries its own path and path_complete inline. And validate no longer issues a second request when the stage is empty: an empty changeset cannot introduce anything, so the baseline is the staged report by construction. That matters because validate is write-scope gated and rate limited per principal, and it was the one case where the extra call bought nothing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
validate's verdict covers the resulting branch, not your change. On a branch that already carries findings — the normal state of a large imported graph —validate && commitis unusable however correct the change is, and a caller cannot tell "I broke this" from "it was already like this".Measured on a real branch: an empty stage returns 99 findings and exit 5.
What it does
Both directions verified against the live service.
Fail-closed, because the errors aren't symmetric
So anything ambiguous is classified introduced.
Identity
(code, raw locator), keyed before path resolution — resolution is presentation, can be partial, and a stale index can fold two ids onto one path.Counted as a multiset, not a set. A locator legitimately carries more than one finding, so with a set a locator going from one finding to two registers as no change — hiding the new one, in the silent direction. Mutation-verified: switching to set semantics fails 7 tests.
The two calls must describe one branch
If the branch moved between the reads, the answers describe different graphs, and attributing across them would blame your stage for someone else's commit — or hide one of yours behind theirs. It refuses to partition and says so. A conservation check on the buckets does the same for a bug in this code. Mutation-verified: removing the version guard fails 2 tests.
JSON
Three buckets as separate arrays rather than one list with a flag, and
validis redefined to match the exit code, with the server's whole-branch answer underwhole_branch_valid. Otherwise the most gateable field in the payload would disagree with the exit status beside it.Default behaviour is unchanged — this ships behind the flag. Flipping the default is the follow-up, and that one changes what exit
5means for existing callers.419 tests.
fmt,clippy -D warningsclean.