fix(ci): a security failure DELETED the test tier (#194); the coverage threshold never gated (#189)

[rubenvdlinde]

Closes #194. Closes #189.

Both defects live in .github/workflows/quality.yml, so they land together.

---

#194 — gating the test tier on the security tier deletes all test evidence

Cause

Four jobs carried the same clause:

if: ${{ … && !cancelled() && needs.security.result != 'failure' }}

phpunit, newman, playwright, journeydoc-capture. (The issue also cites sbom; main had already dropped it there. Verified — 4 sites, not 5.)

composer audit queries the live Packagist advisory feed on every run. So on 2026-08-06, when CVE-2026-67434 began being served against squizlabs/php_codesniffer, the entire fleet's test tier turned skipped simultaneously, with no commit in any repo. The dependency is a code formatter that never executes in production.

The damage was not the skip. It was that a skipped job renders as a grey tick, not a red X — skipped is not failure, so contains(needs.*.result, 'failure') reads straight past it. The absence of a verdict was indistinguishable from a good one, and a review filed a "fully green" report over 16 repos with no test coverage.

Fix — option E (A + D)

(A) Decouple. The four jobs no longer read needs.security.result. needs: is kept for ordering only; the existing !cancelled() already suppresses the implicit success(), so they reach their own verdict regardless.

Security still blocks the merge, unchanged. security is in Quality Report's needs:, and quality / Quality Report is a required context on main and beta in all 25 fleet repos. Nothing is weakened at the merge gate. The cost is a few CI minutes on a PR that could not have merged anyway.

(D) Loud rendering. Two additions to the Quality Report:

• the test rows render an enabled-but-skipped job as 🚨 **NO VERDICT — enabled but never ran**, not ⏭️ — pairing needs.<job>.result with the input that switches the job on, so the 20-odd repos that run no E2E stay quiet;
• a new gate step, "a test job that was ENABLED but never ran has no verdict", which hard-fails with TEST TIER NOT EXECUTED — NO VERDICT EXISTS, names the jobs, and states in words that this is not a test failure but the absence of a test result. When the tests did run and only security failed, it emits a ::notice:: saying exactly that — the distinction the 2026-08-06 reports could not make.

It reuses the superseded-run discriminator from the cancelled gate, so cancel-in-progress does not produce a wall of red.

Evidence — a LIVE Actions run, not a simulation

https://github.com/ConductionNL/.github/actions/runs/31259774225 (fixture/issue-194-evidence-deletion, workflow kept in this PR as test-tier-gating-fixture.yml).

A failing security job and two byte-identical failing test jobs differing only in their if::

job	if:	result
security	—	failure
BEFORE — test gated on security	!cancelled() && needs.security.result != 'failure'	skipped ← evidence deleted
AFTER — test decoupled	!cancelled()	failure ← verdict exists
OLD tally over the test tier	contains(needs.*.result,'failure')	success ← green over a deleted test tier
NEW invariant over the test tier	this PR's gate	failure

Verbatim from the run:

test-gated result: skipped
The old gate is: contains(needs.*.result, 'failure')
'skipped' is not 'failure', so this reads ZERO failures.
CONCLUSION: green. A test suite that would have FAILED was
deleted, and the tally reports nothing wrong.

##[error]TEST TIER NOT EXECUTED — NO VERDICT EXISTS for one or more enabled
test jobs. A skipped test job is not a pass; it is the absence of a result.
Missing: PHPUnit (result=skipped)

The #229 audit, both directions

#229 fixed direction 1 (frontend-tests missing from Quality Report's needs:). Full enumeration of all 17 jobs:

job	needs:	status fn in if:	in report needs:	can a producer DELETE it?
php-quality	—	no	yes	n/a
vue-quality	—	no	yes	n/a
frontend-checks	—	no	yes	n/a
frontend-build	—	no	yes	n/a
frontend-tests	—	no	yes	n/a
security	—	no	yes	n/a
license	—	no	yes	n/a
phpunit	php-quality, security	yes	yes	was YES → no
newman	php-quality, security	yes	yes	was YES → no
playwright	php-quality, security, frontend-build	yes	yes	was YES → no
journeydoc-capture	security	yes	yes	was YES → no
baseline-protection	—	no	yes	n/a
hydra-gates	playwright	yes	yes	no
sbom	security	yes	yes	no
features-check	—	yes	yes	n/a
features-extract	—	yes	yes	n/a
report	(all 16)	always()	is the gate	no

• Direction 1: no holes. All 16 verdict-producing jobs are in needs:; #229 holds.
• Direction 2: 4 holes, all fixed — the four above. No job relies on the implicit success() trap.

scripts/assert-no-producer-deletes-a-verdict.py makes it permanent, checking both shapes (explicit needs.*.result gate; needs: with no status function). Run against origin/main it correctly reports all four. Positive control included.

---

#189 — playwright-coverage-threshold has never gated

Cause, and the decision

Decision: make it gate, and stop lying about what it measures. Removing it was the alternative; the metric is worth having once it means something, and gate-19 already supplies the annotation dialect to make it real.

Three defects, all fixed:

1. It could not fail. ::warning:: + no non-zero exit. Reproduced against the real pipelinq checkout with the verbatim old program: 838 scenarios / 232 tests → 28%, exit 0. Now ::error:: + process.exit(1).
2. It was not coverage. tests.length / scenarios.length — two independent totals never compared to each other. Replaced with real per-scenario matching: a scenario is covered when a Playwright file carries an @e2e reference resolving to its slug, using gate-19's dialect (check_e2e_coverage.py) so a repo annotates once and both numbers move together. @e2e exclude <reason> leaves the denominator. The old ratio survives as testsPerScenarioPercent — reported, never gated on.
   Incidental finding: the old regex /^###?\s+(S\d+|Scenario[:\s]|REQ-)/ cannot match #### Scenario: (four hashes), which is 2659 of pipelinq's headings — the dominant scenario form in the fleet was never counted at all.
3. Zero scenarios scored 100%. Now measurable: false, coverage: null, ::error::NOT MEASURABLE, exit 1 — a failure to measure, not a pass — and it fails even at threshold 0.

gate-19 is not invoked from this job on purpose: it has no hydra-gates checkout, and adding one would make E2E depend on a second repository to report a number. gate-19 stays the diff-scoped PR enforcer; this is the whole-repo floor.

Proof the gate CAN fail

scripts/test-spec-coverage-gate.py extracts the Node program out of this workflow's heredoc and executes it — the shipped text, not a transcription. 24 assertions, all passing, including:

test_below_threshold_fails:
  PASS  below-threshold: exits NON-ZERO (the gate can fail)
  PASS  below-threshold: emits ::error:: not ::warning::
test_zero_scenarios_is_not_a_pass:
  PASS  zero-scenarios: does NOT score 100
  PASS  zero-scenarios: coverage is null, not a number
  PASS  zero-scenarios: fails even at threshold 0
test_unrelated_tests_do_not_raise_coverage:
  PASS  unrelated tests: real coverage is UNCHANGED by 10 noise tests
  PASS  unrelated tests: the old ratio DID move (proving the fixture bites)

Positive control: neuter the gate back to warning-only and 5 assertions flip to FAIL. A suite that stayed green there would not be measuring enforcement.

Both new scripts are wired into quality-resolve-probe.yml, so they run on every PR into main. A checker with no callers is not a checker.

Blast radius — stated, not discovered

Measured 2026-08-08 across all 31 callers of quality.yml in the org (every code-quality.yml, not code search):

repo	enable-playwright-coverage	threshold	honest number today
pipelinq	true	75	22%
hrmq	commented out	—	—
all other 29	unset → false	—	step does not run

pipelinq is the entire blast radius. Its honest number is 22% (2659 scenarios, 2140 @e2e-excluded with reasons, 519 enforceable, 116 covered). At 75 it would go red on merge of this PR with no code change.

So ConductionNL/pipelinq#732 lands first, ratcheting 75 → 20 — the floor it can actually hold, with slack. Nothing turns red on this merge.

The default is also changed 75 → 0, so a repo that switches measurement on gets the number without being turned red by a floor it never chose. Gating is opt-in by setting a value.

⚠️ Residue, stated: pipelinq's beta branch also carries 75 and is human-gated, so it is untouched. It needs the same ratchet on the next development → beta sync or its next beta PR goes red.

---

Files

• .github/workflows/quality.yml — both fixes
• .github/workflows/quality-resolve-probe.yml — wires the two new checkers in
• .github/workflows/test-tier-gating-fixture.yml — new; the quality.yml: gating the test tier on the security tier silently deletes all test evidence (CVE-2026-67434 disabled PHPUnit/Newman/E2E fleet-wide today) #194 live proof, runs only on fixture/**
• scripts/assert-no-producer-deletes-a-verdict.py — new
• scripts/test-spec-coverage-gate.py — new

🤖 Generated with Claude Code

---

Round 2 — mutation testing, and a limitation stated up front

The mutation battery found a real hole in my own test

A single positive control was not enough. --mutation-battery reintroduces seven specific defects one at a time and requires the suite to notice each, plus an anti-widening control that reworks a log string nothing asserts on and which the suite must not notice — without it, a suite that failed on any edit at all would score a perfect kill rate while being worthless.

It earned its keep immediately. On the first run, exclude-directive-read-as-reference SURVIVED:

  SURVIVED exclude-directive-read-as-reference  — SUITE STAYED GREEN.
6 of 7 defect-mutants killed; 1 survived
FAIL — mutant(s) survived. The suite cannot reach that behaviour.

The fixture used @e2e exclude <slug>, space-separated. Under that form the guarded and unguarded regexes are indistinguishable — both capture exclude, which contains no # or :: and so resolves to no slug. Measured:

annotation in a test file	guarded	unguarded
@e2e exclude widget-renders — server side	[]	[] — equivalent
@e2e exclude::widget-renders-on-the-dashboard	[]	["widget-renders-on-the-dashboard"] — differs
@e2e exclude#widget-shows-an-empty-state	[]	["widget-shows-an-empty-state"] — differs

So that assertion had been passing while proving nothing. The guard is load-bearing only for the separator forms, where the unguarded regex silently marks the named scenario covered — the exact opposite of what an exclusion says. Fixture fixed; mutant now dies.

Final: 7 of 7 killed, 0 survived, control correctly survives, 25 assertions at baseline.

An unapplied mutant reports SKIPPED (wiring) and fails the run rather than counting as a kill — a drifted anchor means the battery measures less than it claims.

Known gap, stated rather than discovered later

#249 rewrote gate-19 to read test files with a real JS parser, so it will not count an @e2e reference that sits inside a describe.skip or an empty test body. This step reads the annotation as text and will.

So this number is an UPPER BOUND on real coverage, and the two figures can legitimately disagree in that one direction. Closing the gap means shipping a JS parser into the heredoc; gate-19 already catches the case on every PR that touches a spec. A passing threshold here is not evidence that gate-19 would pass. This is written into the workflow comment, not just here.

Signalling convention

Matches gate-19 post-#249: the exit code is a status only (0/1), never a count — counts go to stdout. A failure to measure is reported distinctly from a measured pass, and an unexpected crash exits non-zero rather than falling through to green.

Housekeeping

• Rebased/merged onto latest main (#217, #246, #249, #248, #250, #255 — all hydra-gates/-only, zero overlap with this branch's five files; verified each file's blob hash is unchanged by the merges).
• run: block size for the new heredoc step: 9,026 bytes against the 16,384 limit (assert-run-steps-resolvable.py green) — nowhere near the #161 resolution-breaking class.