fix(quality): stop the coverage-baseline gate reporting success while doing nothing (#61) - #138
Merged
Merged
Conversation
… doing nothing
`update-baseline` ended in `git push || echo "::warning::..."`. On any repo whose
development/main carries a ruleset ("Changes must be made through a pull request")
that push is rejected, and the `||` swallowed it — so the job reported SUCCESS
while the baseline never moved. Reproduced on openregister run 30851583987, job
91814324437 (conclusion: success):
remote: error: GH013: Repository rule violations found for refs/heads/development.
- Changes must be made through a pull request.
! [remote rejected] development -> development (push declined ...)
A swallowed failure is worse than no job at all: it is indistinguishable from the
work having been done.
Chosen fix: make the gate honest and READ-ONLY, mirroring the design already
shipped for features-extract, rather than granting the bot a ruleset bypass.
Letting CI push to the branch it is judging is the hazard that design deliberately
removed, and a bypass actor would widen what every workflow in the repo can push.
- update-baseline -> read-only (`contents: read`), no commit, no push. Recomputes
the baseline, hard-fails on drift, and attaches the recomputed value as the
`coverage-baseline` artifact. Renamed "Coverage Baseline Check" so the name
stops promising a write.
- baseline-protection -> enforce the property that actually matters (the ratchet
never goes DOWN) instead of banning every manual change. The blanket ban was
coherent only while the bot auto-committed; with a read-only push job it left
the baseline unchangeable by bot or human alike — an unclosable gate. A PR may
now raise the baseline (this is how the push-side gate is closed) but never
lower it. Fails closed on an unparseable value.
- journeydoc-capture -> a refused push now fails the job instead of setting
changed=0, which had made a rejected push look identical to "nothing changed".
Screenshots are still uploaded as an artifact, so a refusal stays recoverable.
Blast radius: `enable-coverage-guard` defaults false and openregister is its only
caller across 22 repos scanned; `enable-journeydoc-capture` has no caller at all.
Nothing `needs:` update-baseline, so it cannot cascade into Quality Report — the
only quality context any ruleset requires.
Refs #61
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes the still-live half of #61.
The bug — a gate that reports success while doing nothing
update-baselineended in:On any repo whose
development/maincarries a ruleset ("Changes must be made through a pull request") that push is rejected, and the||swallows it. The job then reports success.Reproduced on a live run — openregister run 30851583987, job
91814324437, conclusionsuccess:The baseline never moved, and nothing anywhere went red. A swallowed failure is worse than no job at all: it is indistinguishable from the work having been done.
Which fix, and why
Chosen: make the gate honest and read-only — mirroring the design already shipped for
features-extract— rather than granting the bot a ruleset bypass.I have admin on both repos, so adding
github-actions[bot]as a bypass actor was available. I deliberately did not:features-extractredesign removed — a quality pipeline mutating the branch it is judging, via a[skip ci]commit that runs no checks;enable-coverage-guardwould need the same manual ruleset surgery, and forgetting it silently restores the swallowed-push behaviour.Changes
update-baseline→ "Coverage Baseline Check"contents: write, commit +git push || echo ::warning::contents: read, no commit/push. Recomputes, hard-fails on drift, attaches the recomputed value as thecoverage-baselineartifactbaseline-protection.coverage-baselinechangejourneydoc-capture::warning::+changed=0Why
baseline-protectionhad to change tooMaking the push job fail loudly on its own would have produced an unclosable gate:
baseline-protectionforbade humans editing.coverage-baseline, and the bot could not push it — so the baseline would be unchangeable by anyone, and every push todevelopmentwould be permanently red with no way to close it.The blanket ban was only ever coherent while the bot auto-committed. The anti-gaming property actually worth enforcing is the baseline never goes down —
coverage-guard.phponly writes it whencurrent > baseline. So a PR may now raise it (that is how the push-side gate is closed) but never lower it.Verification
The new
baseline-protectionscript was extracted verbatim from the YAML and run against real git repos. Every failing case was shown to actually fail:.coverage-baselinechange55.54 → 60.0055.54 → 50.0055.54 → 55.54abc0.019 → 1010 → 9Differential against the logic shipping today, confirming this is not a no-op: old logic rejects
raised 55.54 → 60.00(the deadlock); new logic accepts it while still rejecting every lowering.An earlier version of that harness was itself broken (
git init -bis unsupported on git 2.25.1), so every case fell through to "no baseline change" and falsely passed. Caught by comparing expected against actual rather than reading the passes.What this change cannot break
Scanned 22 caller repos.
enable-coverage-guarddefaults false; openregister is its only caller. No other repo can reach either baseline job at all.enable-journeydoc-capturedefaults false and no caller in the fleet sets it — 0 executions. That edit cannot execute today.needs:update-baseline(verified by parsing the job graph), so it cannot cascade intoQuality Report— the only quality context any ruleset requires (openregisterBeta/Main).baseline-protectiondoes feedQuality Report, but the new scriptexit 0s early when the PR does not touch.coverage-baseline— every PR today. On the only path it can reach it is strictly more permissive than before.Update Coverage Baseline, and the job runs onpushonly, so it can never be a required PR status check. The job key is unchanged.YAML parses; job count unchanged at 18.