Skip to content

fix(quality): stop a phpmd finding from silently deleting the PHPUnit signal - #140

Merged
rubenvdlinde merged 1 commit into
mainfrom
fix/phpunit-not-gated-on-static-analysis
Aug 4, 2026
Merged

fix(quality): stop a phpmd finding from silently deleting the PHPUnit signal#140
rubenvdlinde merged 1 commit into
mainfrom
fix/phpunit-not-gated-on-static-analysis

Conversation

@rubenvdlinde

Copy link
Copy Markdown
Contributor

The defect

phpunit was gated on needs.php-quality.result != 'failure'.

php-quality is a matrix job, so its result is failure if any leg fails — lint, phpcs, phpmd, psalm, phpstan, phpmetrics. A complexity or formatting finding therefore deleted the entire PHPUnit matrix.

This is a dead gate in the strict sense: its absence looks exactly like its success. A skipped matrix renders in the checks list as a single greyed-out skipping placeholder — not red, not missing, just quietly not there. Nothing in the PR view says "your tests did not run".

The evidence — a natural A/B, measured 2026-08-04

repo phpmd job total jobs PHPUnit legs
ConductionNL/procest (run 30863839239) failure (25 findings) 26 0
ConductionNL/decidesk (run 30862373575) success 29 4, all green

Identical workflow inputs. The only difference is phpmd's colour.

Two things follow:

  1. procest's 1684 tests had no CI signal at all for as long as phpmd was red.
  2. decidesk's phpmd was green only because phpmd.baseline.xml was suppressing 73 real findings. So whether decidesk's suite ran at all was contingent on a suppression file. Delete the baseline honestly — which is the direction that repo is going — and its test signal would have gone dark too.

That second point is the one that makes this structural rather than cosmetic: the fix for one problem (remove the suppression) silently caused another (lose the tests).

The change

-    if: ${{ inputs.enable-php && inputs.enable-phpunit && !cancelled() && needs.php-quality.result != 'failure' && needs.security.result != 'failure' }}
+    if: ${{ inputs.enable-php && inputs.enable-phpunit && !cancelled() && needs.security.result != 'failure' }}
  • security gate kept. A repo with a known-vulnerable dependency should not spin up servers.
  • needs: retained for ordering; with !cancelled() and no result condition, PHPUnit runs regardless of static-analysis colour.
  • This aligns phpunit with playwright, which in this same file already deliberately declines to gate on php-quality (its if: checks only needs.security.result). That inconsistency is itself evidence the php-quality gate on phpunit was an oversight rather than a decision.
  • newman keeps its php-quality gate. It stands up a live stack and is a separate cost/risk decision; left unchanged and noted inline.

Verification

  • YAML parses; jobs count unchanged at 18; phpunit.if and playwright.if are now equivalent in their php-quality treatment.
  • A live positive control is running: ConductionNL/procest@ci/prove-phpunit-gate points code-quality.yml at this branch while procest's phpmd is still red. If the fix works, that run shows phpmd failing and the PHPUnit legs executing. Result posted below as a comment.

Blast radius

Fleet-wide (every Conduction app consumes quality.yml@main). The effect is strictly more signal: repos whose php-quality is green see no change; repos whose php-quality is red gain back their test results. CI cost rises only for repos that are currently red.

… signal

`phpunit` was gated on `needs.php-quality.result != 'failure'`. php-quality is
a matrix job, so its result is 'failure' if ANY leg fails — including phpmd
(complexity/mess detection) and phpcs (formatting). Neither tells you anything
about whether the tests pass, but a red leg deleted the entire PHPUnit matrix.

The loss is invisible by construction: a skipped matrix renders as one greyed
"skipping" placeholder in the checks list, which reads like a non-event rather
than a gate that stopped running.

Measured 2026-08-04:

  ConductionNL/procest   phpmd RED (25 findings)  → 26 jobs, ZERO PHPUnit legs
  ConductionNL/decidesk  phpmd green              → 29 jobs, 4 green PHPUnit legs

Same workflow inputs; the only difference was phpmd's colour. procest's 1684
tests had had no CI signal at all for as long as phpmd had been red. And
decidesk's phpmd was green only because a phpmd.baseline.xml was suppressing 73
real findings — so whether its suite ran at all was contingent on a suppression
file.

The `security` gate is kept: a repo with a known-vulnerable dependency should
not spin up servers. This aligns phpunit with the playwright job, which already
deliberately declines to gate on php-quality. `needs:` is retained for ordering.

Newman keeps its php-quality gate for now — it stands up a live stack and is a
separate risk/cost decision; noted inline.
@rubenvdlinde

Copy link
Copy Markdown
Contributor Author

Live positive control — the fix works, and it immediately found something

ConductionNL/procest@feature/prove-phpunit-gate is development with one line changed: code-quality.yml points at this branch instead of @main. procest's phpmd is still red (25 findings), so under the old gate the PHPUnit matrix could not run.

Run 30866791400:

before (run 30863839239, @main) after (run 30866791400, this branch)
PHP Quality (phpmd) failure failure (unchanged — still 25 findings)
PHPUnit legs 0 (single skipping placeholder) 4 — PHP 8.3/8.4 × NC stable31/stable32
total jobs 26 29

phpmd red, PHPUnit ran. That is exactly the behaviour the old if: made impossible.

What it surfaced — the gate was hiding a second, worse defect

All four PHPUnit legs failed:

Tests: 1684, Assertions: 5614, Skipped: 5, Risky: 223.
Process completed with exit code 1

Zero failures, zero errors — 223 risky tests. procest's phpunit.xml sets failOnRisky="true" and beStrictAboutCoverageMetadata="true", and CI runs phpunit --coverage-clover, so every test that touches a class not named in its #[CoversClass]/#[UsesClass] list is risky and the run exits 1:

1) ...AssistantControllerTest::testHermiqGuardrailBlockRelaysStatusAndErrorCode
This test executed code that is not listed as code to be covered or used:
- OCA\Procest\Service\Assistant\HermiqAssistantException

This is invisible locally: composer test:all runs without coverage, so it reports 1684 passing and exits 0. Only the CI invocation collects coverage, and that invocation had never once been allowed to run.

So procest's PHPUnit job was not merely dark — it was red on arrival. Two dead-gate causes stacked: the job was skipped, and had it not been skipped it could never have passed. The skip is what kept the second one invisible.

For contrast, decidesk carries identical phpunit.xml strictness (failOnRisky="true", beStrictAboutCoverageMetadata="true") and its 4 legs are green — so this is procest's annotations being incomplete, not a bad setting to be loosened.

Consequence for merging this PR

Merging turns procest's PHPUnit red where it is currently silent. That is the correct outcome and the entire point — but it should land together with, or just ahead of, the annotation fix, so procest is not left red on a newly-visible axis. Tracking that separately; this PR is what makes the defect visible at all.

The probe branch feature/prove-phpunit-gate will be deleted once this merges.

@rubenvdlinde
rubenvdlinde merged commit 9d6ffef into main Aug 4, 2026
4 checks passed
rubenvdlinde pushed a commit to ConductionNL/procest that referenced this pull request Aug 4, 2026
…risky → 0 under coverage

procest's PHPUnit legs had never run in CI. They were gated on
`needs.php-quality.result != 'failure'`, and procest's phpmd leg has been red
on 25 complexity findings, so `development` runs produced 26 jobs with a single
greyed-out PHPUnit placeholder. ConductionNL/.github#140 removed that gate; the
matrix expanded to 29 jobs and the suite failed on its first real run.

The failure was invisible locally. `composer test:all` runs PHPUnit WITHOUT
coverage, and all 1684 tests pass that way. CI runs `--coverage-clover`, and
phpunit.xml sets `beStrictAboutCoverageMetadata="true"` + `failOnRisky="true"`.
That combination only has teeth when a coverage driver is present: with covers
metadata declared, PHPUnit marks a test risky if it executes production code
that is listed neither as covered nor as used. 220 tests did.

Every one of the 220 reported the same reason — "This test executed code that
is not listed as code to be covered or used". None of them were missing
`@covers`; they were exercising a collaborator the class docblock never
declared: result value objects (ActionResult, GuardResult, BagLookupResult),
domain exceptions (DecisionEvaluationException, HermiqAssistantException), the
IntegrationMode enum, the SearchesObjects trait, and a handful of base classes
and helper services.

`@uses` is the correct declaration for all of them: it permits the collateral
execution without attributing the code as covered, so reported coverage is
unchanged (30.87% line coverage; the guard's baseline is 0.00 and passes).
Nothing here weakens a check — `failOnRisky`, `beStrictAboutCoverageMetadata`
and `--coverage-clover` are all untouched, and no test assertion changed.

The risky set is execution-order-dependent: for a class whose only executed
lines are load-time (enum cases, class constants), the attribution lands on
whichever test loads it first, so consecutive local runs reported 227 and then
220. The declarations here were derived from the union of a full-suite run and
a 245-file isolated sweep, then verified both ways: full suite Risky: 0 on two
consecutive fresh-cache runs, and 0 risky pairs across all 245 files run in
isolation.

Tests: 1684, Assertions: 5614, Skipped: 5, Risky: 220 → 0. Exit 1 → 0.
rubenvdlinde added a commit to ConductionNL/procest that referenced this pull request Aug 4, 2026
… 0 (#716)

procest's PHPUnit legs had never run in CI: they were gated on `needs.php-quality.result != 'failure'` and phpmd has been red, so development produced 26 jobs with one greyed-out placeholder. ConductionNL/.github#140 removed that gate and the suite failed on its first real execution with 223 risky tests.

All 223 reported the same reason — 'executed code that is not listed as code to be covered or used'. None were missing @Covers; each was executing a collaborator its docblock never declared (result value objects, domain exceptions, the IntegrationMode enum, the SearchesObjects trait, base classes). Declared with @uses, which permits the collateral execution without attributing it as covered.

failOnRisky, beStrictAboutCoverageMetadata and --coverage-clover are untouched; no assertion, fixture or lib/ file changed. 87 files, +249 lines, all docblock annotation under tests/Unit/.

Verified: Tests: 1686, Assertions: 5632, Skipped: 5, Risky: 0 on all four legs (PHP 8.3/8.4 x NC stable31/stable32). Positive control pushed and reverted — a bogus assertion plus one removed @uses turned all four legs red naming both, and the revert was confirmed byte-identical to the pre-control tree.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant