fix(ci): compare the coverage ratchet against the measured merge base

[rubenvdlinde]

The coverage ratchet asks for a number nobody can compute

.coverage-baseline is read two ways at once:

• the phpunit guard fails when coverage is below it;
• the push-side Coverage Baseline Check fails when the file is stale — that is, when coverage is above it.

Together those demand exact equality with a checked-in constant. The only way to restore that equality is to commit, in a pull request, the number the tree will measure after that pull request lands — while the base branch keeps moving and your own diff changes the answer.

Measured on openregister: an author committed 58.93; development advanced 16030 → 16038 tests while the PR sat; the merge result measured 58.88; the guard reported "dropped by 0.05%". While the follow-up pinned 58.88, development moved two further commits. There is no value a PR author can commit that is guaranteed correct when it lands.

It is not one repo. On 2026-08-06, six of the sixteen apps were red on this job, every one of them for coverage being higher than recorded:

repo	committed	CI measured	delta
openregister	58.87	58.88	+0.01
docudesk	61.39	61.41	+0.02
procest	29.60	29.69	+0.09
openbuild	57.39	58.83	+1.44
larpingapp	67.68	71.73	+4.05
doriath	55.78	55.79	+0.01

Three of those reds are one or two hundredths of a percent — a single extra covered statement.

A third trap sits underneath: CI measures with xdebug, local runs typically use pcov, and the two do not count statements identically. A baseline committed from a local measurement can be wrong on arrival.

The fix: measure the floor, don't type it

scripts/coverage-guard.php gains --against=<clover.xml>, naming a coverage report measured at the merge base. When it is given it is the only floor — the committed .coverage-baseline is printed for information and deliberately not enforced.

That precedence is the point. Taking max(committed, measured) would preserve the openregister failure exactly, so the measured value does not merely win ties, it replaces the constant outright. Because both numbers then come from the same driver in the same job, the xdebug/pcov difference cancels instead of being baked into a constant. And the merge base is immutable for a given head, so the comparison cannot go stale: a rebase re-measures both sides together.

The companion change in ConductionNL/.github measures the merge base in the phpunit job and retires the push-side staleness job. That change lands after this one, and it probes --capabilities and fails loudly rather than silently falling back — an older copy of this script accepts --against and ignores it, which would demote the ratchet to the constant check while still reporting success.

Without --against (pushes to main/development) the committed .coverage-baseline is still enforced, but strictly as a floor: below it fails, above it is fine. Demanding equality is what made the gate unsatisfiable.

It still catches a real drop — proof

Verified against two real CI clover artifacts from this fleet (openbuild runs 31016006546 → 8018/13971 = 57.39% and 31050116167 → 8229/13987 = 58.83%), not synthetic input:

case	result
real 1.44% drop (head=57.39 vs base=58.83)	exit 1 — FAIL
improvement (head=58.83 vs base=57.39)	exit 0
unchanged	exit 0
PR adds 20 untested statements	exit 1 — FAIL
PR adds 20 tested statements	exit 0
one covered statement lost (8229 → 8228)	exit 1 — FAIL
empty merge-base report (0 statements)	exit 2 — refuses, does not read as 0%
missing merge-base report	exit 2 — refuses
unparseable committed floor	exit 2 — fails closed

Two of those cases are holes found in the first draft of this change and closed before it shipped:

• A one-statement regression rounded away. At two decimals 8229/13987 and 8228/13987 both read 58.83%, so the guard printed two different statement counts next to the word "unchanged" and exited 0. Comparison is now an exact integer cross-product of the ratios, not a comparison of rounded percentages. One statement is a small hole, but it does not close by itself: a regression that gives back a statement at a time is invisible for as many PRs as it cares to take.
• An empty clover report is now a hard error, not 0%. As the merge-base side it would otherwise set the floor to zero and pass every possible drop.

And the load-bearing comparison — the old guard passes that same real 1.44% drop:

$ php scripts/coverage-guard.php old-clover.xml     # committed floor 57.39
Coverage baseline: 57.39%
Coverage current:  57.39%
Coverage unchanged.
>>> EXIT=0

openbuild's floor had decayed so far below actual coverage that a regression all the way back to 57.39% was invisible to it. The new comparison fails it. This change makes the ratchet stricter, not laxer.

What is deliberately not changed

.coverage-baseline values are left alone. Every repo's committed floor is currently at or below its measured coverage, so none is wrong in the dangerous direction, and pinning each to today's exact measurement would put the push-side check back on a knife edge — the brittleness this change exists to remove. A floor below actual coverage is conservative, never wrong, and is never the binding constraint on a PR. It may be raised at any time; the Coverage Baseline Protection job still refuses to let it be lowered.

No waiver, no continue-on-error, no widened threshold.

[github-actions]

Quality Report — ConductionNL/nldesign @ 95859fc

Check	PHP	Vue	Security	License	Tests
lint	✅
phpcs	✅
phpmd	✅
psalm	✅
phpstan	✅
phpmetrics	✅
eslint		✅
stylelint		✅
build		✅
check-manifest		✅
test-l10n		✅
composer			✅	✅ 100/100
npm			✅	✅ 2/2
PHPUnit					✅
Newman					❌
Playwright					✅
Hydra gates					❌

---

Quality workflow — 2026-08-06 06:32 UTC

Download the full PDF report from the workflow artifacts.