test(e2e): stop 18 backend-only scenarios laundering gate-19 coverage through an always-skipped test

[rubenvdlinde]

The finding

tests/e2e/spec-coverage/document-zaakdossier.spec.ts contained a test tagged with 18 @e2e scenario anchors whose body ended in:

test.skip(true, 'Backend-enforced — asserted in PHPUnit/Newman, not UI')

Unconditional. It reported skipped on every run, in every environment, while still supplying gate-19 with the traceability link for all 18 scenarios. REQ-ZAK-001a–c, 002a–d, 003a–d, 007a–b, 008b, 009a–b and 010a–b were recorded as e2e-covered by a test that never executed.

The stated reason was honest about why the test could not assert anything — but the mechanism laundered coverage: the gate cannot tell a tag on a running test from a tag on a permanently skipped one.

The fix

Those 18 scenarios really are backend-only (service guards, HTTP status contracts, Content-Range framing, an info.xml repair step). So their traceability moves to the mechanism gate-19 provides for exactly this case — a reason-bearing @e2e exclude on each scenario in openspec/specs/document-zaakdossier/spec.md, each naming the PHPUnit or Newman suite that does assert it. The placeholder test is deleted.

Three further tests ended in an unconditional test.skip(true, 'Requires a seeded case fixture …'). The blocker is real, but permanent — nothing in this repo seeds a dossier fixture for Playwright, so the condition can never become true. They become test.fixme referencing #764, so the quarantine is visible in the declaration rather than hidden behind a skip that reads as transient.

Coverage accounting

	before	after
scenarios traced by a running test	2	2
scenarios traced by a skipped/fixme test	26	8 (all fixme, all → #764)
scenarios with a written @e2e exclude reason	0	18
scenarios with no traceability at all	0	0

Verified by partition: the 18 excluded and the 10 still tagged by the spec file are disjoint and together cover every #### Scenario in the spec — no scenario is left uncovered and none is double-counted.

Note on the wider inventory

This came out of a fleet-wide triage of disabled e2e declarations. Worth recording: the great majority of test.skip(true, …) calls across the fleet are inside if/else/catch guards — the literal true is Playwright's "skip from this point" API shape, not evidence that the skip is unconditional. Across 11 repos only 4 unconditional test.skip(true) calls existed, and all 4 were in this one file.

Closes nothing on its own; #764 tracks the seeding work that would let the three fixme tests run.

[github-actions]

Quality Report — ConductionNL/procest @ 9b9991b

Check	PHP	Vue	Security	License	Tests
lint	✅
phpcs	✅
phpmd	✅
psalm	✅
phpstan	✅
phpmetrics	✅
eslint		✅
stylelint		✅
build		✅
check-manifest		✅
check-vue3-compile		✅
test-l10n		✅
composer			✅	✅ 100/100
npm			✅	✅ 552/552
PHPUnit					✅
Newman					⏭️
Playwright					✅
Hydra gates					❌

---

Quality workflow — 2026-08-08 10:46 UTC

Download the full PDF report from the workflow artifacts.

[github-actions]

Quality Report — ConductionNL/procest @ 303483c

Check	PHP	Vue	Security	License	Tests
lint	✅
phpcs	✅
phpmd	✅
psalm	✅
phpstan	✅
phpmetrics	✅
eslint		✅
stylelint		✅
build		✅
check-manifest		✅
check-vue3-compile		✅
test-l10n		✅
composer			✅	✅ 100/100
npm			✅	✅ 552/552
PHPUnit					✅
Newman					⏭️
Playwright					❌
Hydra gates					❌

---

Quality workflow — 2026-08-08 11:10 UTC

Download the full PDF report from the workflow artifacts.

[github-actions]

Quality Report — ConductionNL/procest @ 1805d49

Check	PHP	Vue	Security	License	Tests
lint	✅
phpcs	✅
phpmd	✅
psalm	✅
phpstan	✅
phpmetrics	✅
eslint		✅
stylelint		✅
build		✅
check-manifest		✅
check-vue3-compile		✅
test-l10n		✅
composer			✅	✅ 100/100
npm			✅	✅ 552/552
PHPUnit					✅
Newman					⏭️
Playwright					✅
Hydra gates					❌

---

Quality workflow — 2026-08-08 11:33 UTC

Download the full PDF report from the workflow artifacts.

[rubenvdlinde]

Verification

E2E Tests (Playwright) — success:

	tests	passed	failed	skipped
baseline (last procest pull_request run with an E2E job, fix/coverage-baseline-29-69)	126	87	0	39
this branch	125	87	0	38

Total −1 (the deleted placeholder test), skipped −1, passed unchanged. Note the dependabot PR run has no E2E job at all, so it is not a usable baseline.

Positive control

A sentinel spec was pushed that boots the real app through the same navigation and asserts an impossible string. With it present the E2E job reported 1 failed / 87 passed / 38 skipped and concluded failure; it was then removed and the job returned to 87 passed / 0 failed / 38 skipped. So the green above is a real verdict, not a job that cannot go red.

Hydra Gates is red — pre-existing, and gate-19's remaining 10 are explained

quality / Hydra Gates fails on [gate-19] e2e-coverage: FAIL — 10 scenario(s). Two things about that:

1. gate-19 was already red on procest PRs before this change. The two most recent baseline PR runs (fix/coverage-baseline-29-69, fix/wire-bewijsstuk-immutability-guard — neither touching this spec) both report [gate-19] e2e-coverage: FAIL — 30 scenario(s) missing @e2e.

2. On this spec's own scope, this change takes gate-19 from 28 findings to 10. Measured by running check_e2e_coverage.py over the origin/development copies of openspec/specs/document-zaakdossier/spec.md + tests/e2e/spec-coverage/document-zaakdossier.spec.ts with all 28 scenarios in the diff:

BEFORE (origin/development): [gate-19] FAIL — 28 scenario(s) without a running e2e test
AFTER  (this branch):        [gate-19] FAIL — 10 scenario(s) without a running e2e test

Every one of the spec's 28 scenarios was previously flagged. The gate had already worked out that this spec's e2e coverage was fictitious; nothing surfaced it because gate-19 is diff-scoped and no PR had touched the spec.

The remaining 10, split honestly

• 8 are gate-19 being correct — req-zak-004c, 005a/b/c, 006a/b, 008a/c are on test.fixme tests blocked by e2e: zaakdossier UI scenarios are permanently skipped — no seeded case fixture in the Playwright environment #764. They genuinely do not run. @e2e exclude would be the wrong remedy: it means "out of scope for e2e, forever", and these should be tested once a fixture exists. gate-19 has no state for "blocked on a tracked issue" — raised as a secondary point in gate-19: a CONDITIONAL test.skip(true, reason) inside an if guard is read as a permanent skip — a test that ran and PASSED is reported as "never runs" .github#239.
• 2 are a gate-19 false positive — req-zak-004a/004b are tagged on test('cases index renders so a case dossier tab can be opened'), which carries no .skip/.fixme modifier. Its only skip is test.skip(true, 'Procest dev container not reachable') inside if (!response). gate-19's discriminator is the argument rather than the call site, so a guarded conditional skip reads as permanent. That test is one of the 87 that passed on this very commit.

Reproduced minimally (two tests, one file, identical but for a guarded skip — the guarded one fails, the clean one passes) and filed as ConductionNL/.github#239, with the fleet blast radius: 111 guarded occurrences of this idiom across openregister/procest/hermiq/docudesk/nldesign versus 4 genuinely unconditional ones.

I have deliberately not added @e2e exclude to any of the 10. For the 2 false positives that would delete a true coverage claim and permanently mark a tested scenario untestable; for the 8 it would misdescribe a tracked fixture gap as a permanent exclusion.