Skip to content

fix(quality): two dead gates in the coverage ratchet — an artifact that never existed, and a job deleted by an unrelated failure - #156

Open
rubenvdlinde wants to merge 2 commits into
mainfrom
fix/coverage-baseline-artifact-hidden-file
Open

fix(quality): two dead gates in the coverage ratchet — an artifact that never existed, and a job deleted by an unrelated failure#156
rubenvdlinde wants to merge 2 commits into
mainfrom
fix/coverage-baseline-artifact-hidden-file

Conversation

@rubenvdlinde

@rubenvdlinde rubenvdlinde commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

The bug

update-baseline ("Coverage Baseline Check") hard-fails when measured coverage
has drifted past the committed baseline, and its error tells you how to close
the gate:

Commit the recomputed value (attached as the coverage-baseline artifact of
this run
) in a pull request.

That artifact has never existed. .coverage-baseline is a dotfile, and
actions/upload-artifact@v4 excludes hidden files by default
(include-hidden-files: false). When the glob then matches nothing the action
does not fail — it prints a warning and the step still succeeds:

##[warning]No files were found with the provided path: .coverage-baseline.
No artifacts will be uploaded.

So the gate instructs you to close it using a file it has just silently declined
to produce. That is an unclosable gate, and it is the same failure shape this
job was previously rewritten to remove (the swallowed git push that made a
rejected push read as work having been done).

Reproduced, not inferred

docudesk, run
30925609908,
the first ever execution of this job in that repo. In one job log, in order:

  1. Measured coverage has moved beyond the committed baseline:
  2. git --no-pager diff -- .coverage-baselineprints the diff, proving the
    file is on disk and modified.
  3. ##[warning]No files were found with the provided path: .coverage-baseline.

Step 2 and step 3 are three lines apart and contradict each other. Only the
hidden-file default explains both.

The fix

include-hidden-files: true on that one upload step, with the reasoning recorded
inline so the next person does not have to re-derive it.

Not fixed here, but found while confirming it — please read

Push runs are being cancelled by concurrent pull_request runs on merge into
development, which voids every push-only gate.
Measured across the 22 repos
whose callers I touched today, 5 had their post-merge push run cancelled
within seconds:

repo push run created killed after the PR run that superseded it
portaliq 30925522964 15:42:38 3s 30925524631 @ 15:42:40
pipelinq 30925477226 15:42:06 8s 30925487141 @ 15:42:13
scholiq 30925471117 15:42:02 95s 30925482542 @ 15:42:09
nldesign 30925495834 15:42:19 118s 30925502622 @ 15:42:23
hermiq 30925561067 15:43:05 307s 30925566460 @ 15:43:09

The job cap is 45 minutes, so none of these is a timeout — they are
concurrency: cancel-in-progress kills. And it is not new: hermiq's 14:31 push
run and nldesign's 12:55 push run were cancelled the same way, before any change
of mine.

This matters more than it looks, because SBOM, Features Extract and
Coverage Baseline Check only ever run on push
. When the push run dies, those
three reach no verdict at all — and a cancelled run is not a red, so nothing
announces it. It is the same "absence looks like success" problem the opt-in
gates had, one level up.

I have not changed the concurrency block here: it is fleet-wide, several
callers override it with their own group expression, and getting it wrong would
be worse than the current state. Flagging it with the measurements rather than
guessing at a fix.


Second bug, found while confirming the first: a failing PHPMD deletes the coverage ratchet

update-baseline's condition carried no status function:

if: ${{ inputs.enable-coverage-guard && github.event_name == 'push' && (...) }}
needs: phpunit

A job condition with no status function is implicitly wrapped in success(),
and that success() is not satisfied by the declared dependency alone — a
failure anywhere in the needs closure removes the job. phpunit sits behind
needs: [php-quality, security] and survives a red php-quality only
because its own if carries !cancelled(). This job had no such escape, so a
failing PHPMD — a grandparent it does not depend on — silently deleted the
coverage ratchet while phpunit itself was green.

Measured, on procest's own push history

run php-quality phpunit Coverage Baseline Check
30896879219 FAIL success skipped
30895657851 FAIL success skipped
30887852492 FAIL success skipped
30882883115 FAIL success skipped
30876345083 FAIL failure skipped
30903829414 ok success success
30902997025 ok success failure (a real verdict)

Five consecutive runs in which the ratchet reported nothing, in a repo that has
had enable-coverage-guard: true all along — and nothing announced it, because
a skipped job is not a red. Reproduced on doriath twice today (run
30925445843 and its re-run): steps: [], started_at == completed_at, the shape
of a job that never existed.

Fix

Add !cancelled() && needs.phpunit.result == 'success', mirroring what phpunit
already does one level up. It also makes the intent legible: this job needs a
successful phpunit because it downloads that job's coverage artifact. It has
no opinion about PHPMD.

Why this matters beyond one job

Both bugs in this PR have the same signature as the ones #147/#148/#149/#150
addressed: the check's absence is indistinguishable from its success. One
produced an artifact that never existed; the other removed a job entirely. In
neither case did anything go red.

…er uploaded

update-baseline tells you to close the gate by committing 'the recomputed
value attached as the coverage-baseline artifact of this run'. That
artifact has never existed: .coverage-baseline is a dotfile and
actions/upload-artifact@v4 excludes hidden files by default, then warns
rather than fails when the glob matches nothing. The gate asked you to
close it with a file it had silently declined to produce.

Reproduced on docudesk run 30925609908, where the preceding git diff
prints the changed file three lines before the upload reports it missing.
update-baseline's if carried no status function, so GitHub wrapped it in
an implicit success() over the whole needs closure. phpunit survives a red
php-quality because its own if has !cancelled(); update-baseline had no
such escape, so a failing PHPMD - a grandparent it does not depend on -
removed the coverage ratchet while phpunit itself was green.

Measured on procest: five consecutive push runs on development where
php-quality failed, phpunit succeeded, and the ratchet reported nothing.
Reproduced twice on doriath today.

Adds !cancelled() && needs.phpunit.result == 'success', mirroring phpunit.
@rubenvdlinde rubenvdlinde changed the title fix(quality): coverage-baseline artifact is a dotfile and was never uploaded (unclosable gate) fix(quality): two dead gates in the coverage ratchet — an artifact that never existed, and a job deleted by an unrelated failure Aug 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant