fix(judge): treat incomplete tool analysis as a warning, not a gate failure - #23
Merged
Merged
Conversation
…ailure tool_execution collapsed two different outcomes into one FAIL: a tool that could not run at all, and a tool that ran but could only analyze part of the target. The second is the common case for Error Prone on any repo using annotation processors (Lombok, AutoValue, Dagger…): javac hits unresolved symbols, Error Prone reports what it could and flags the run incomplete — and gavel failed the verdict of an otherwise-clean repo for a reason that has nothing to do with code quality. Split the two. A Failure now carries a degraded flag: hard failures (executionSuccessful=false) still fail the gate; degraded runs — surfaced by a tool as warning-level toolExecutionNotifications on a *successful* invocation — pass the gate but the ruling carries an honest "incomplete analysis — …" detail so the gap is never silent. Same principle as the TypeScript unanalyzed-tools warning: never fail on a legitimate input, never hide that coverage was partial. The wrapper side that emits the degraded signal ships in gavel_tools; this change is backward-compatible with the current wrapper (executionSuccessful=false keeps failing) and starts honoring degraded runs the moment the new wrapper lands.
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
Gavel verdict
|
JorgeOlmosDev
added a commit
that referenced
this pull request
Jul 13, 2026
…ailure (#23) * fix(judge): treat incomplete tool analysis as a warning, not a gate failure tool_execution collapsed two different outcomes into one FAIL: a tool that could not run at all, and a tool that ran but could only analyze part of the target. The second is the common case for Error Prone on any repo using annotation processors (Lombok, AutoValue, Dagger…): javac hits unresolved symbols, Error Prone reports what it could and flags the run incomplete — and gavel failed the verdict of an otherwise-clean repo for a reason that has nothing to do with code quality. Split the two. A Failure now carries a degraded flag: hard failures (executionSuccessful=false) still fail the gate; degraded runs — surfaced by a tool as warning-level toolExecutionNotifications on a *successful* invocation — pass the gate but the ruling carries an honest "incomplete analysis — …" detail so the gap is never silent. Same principle as the TypeScript unanalyzed-tools warning: never fail on a legitimate input, never hide that coverage was partial. The wrapper side that emits the degraded signal ships in gavel_tools; this change is backward-compatible with the current wrapper (executionSuccessful=false keeps failing) and starts honoring degraded runs the moment the new wrapper lands. * build(deps): bump gavel_tools to 0.3.8 for degraded-analysis wrapper
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
tool_executioncollapsed two very different outcomes into a single FAIL:The second is the common case for Error Prone on any repo using annotation processors (Lombok, AutoValue, Dagger, MapStruct…): its isolated javac hits unresolved symbols, Error Prone reports what it could and flags the run incomplete. gavel then failed the verdict of an otherwise-clean repo for a reason unrelated to code quality — observed on buildfarm (Lombok), where
tool_executioncame back FAIL.Same class of bug as the TypeScript false-green we just fixed: gavel giving a wrong verdict on a legitimate input.
Fix
Split the two outcomes. A
toolexecution.Failurenow carries adegradedflag:executionSuccessful=false) → still fails the gate.toolExecutionNotificationson a successful invocation) → the gate passes, but thetool_executionruling carries an honestincomplete analysis — …detail so the gap is never silent.Never fail on a legitimate input; never hide that coverage was partial.
Scope / compatibility
Failure+ ruling + DTO + SARIF parser).executionSuccessful=falsekeeps failing. It starts honoring degraded runs the moment the matchinggavel_toolswrapper (emitsexecutionSuccessful=true+ warning notification for the incomplete case) lands and the pin is bumped.Tests (TDD)
incomplete analysisdetail; a hard failure alongside a degraded one still FAILs.