Skip to content

test(e2e): 21 real Playwright tests + 4 exclusions close gate-19 137 -> 112 - #486

Merged
rubenvdlinde merged 3 commits into
developmentfrom
fix/gate19-e2e-coverage
Aug 11, 2026
Merged

test(e2e): 21 real Playwright tests + 4 exclusions close gate-19 137 -> 112#486
rubenvdlinde merged 3 commits into
developmentfrom
fix/gate19-e2e-coverage

Conversation

@rubenvdlinde

@rubenvdlinde rubenvdlinde commented Aug 11, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

What this is

gate-19 (e2e-coverage) was softwarecatalog's only failing gate. This
closes a quarter of it with real Playwright tests, not with a lower number.
start 137
end 111
closed by real tests 26 scenarios (19 test cases)
closed by exclusions 0 — see below

Baseline reproduced locally byte-identical to CI using the canonical
gate package (freshly cloned .github@main, 1735 lines), scoped
HYDRA_GATE_BASE_REF=origin/beta — what CI resolves for the open
development → beta PR #197 (run 31478267110).

⚠️ Do not read this PR's own gate-19 as the result

This PR targets development, so gate-19 here runs against a diff containing
one spec file and will pass over a nearly-empty scope. That green is not
the win. The number that matters is the development → beta scope above, and
it is 112, not 0.

Negative control

Run on the final tree, canonical checker:

step count
with suite-wizard.spec.ts 112
file removed 118 (+6, exactly its 6 @e2e anchors)
file restored 112

Every test was proven able to fail

Each of the 14 got a planted true positive, chosen so that only its own test
went red — applicaties: [] in buildSuitePayload reddened only the submit
test; renaming the third wizard step reddened only the step-labels test. All
plants removed and verified absent from source and from the built bundle.

Two tests needed the plant to remove two layers: Nextcloud core's
SecurityMiddleware rejects an unauthenticated request before the controller
body runs, so the anon-POST test only went red once #[PublicPage] +
#[NoCSRFRequired] were added as well. Worth knowing generally — a plant
inside the controller cannot prove your auth test works.

Full suite on an isolated single-owner rig: 90 passed, 4 skipped, 0 failed.
The 4 skipped are pre-existing test.fixme cases, untouched.

What the tests found — all filed, none worked around

#484 and #485 are deliberately not touched here: converting positive
coverage claims into exclusions inside a PR that lowers the gate count is
indistinguishable from gaming it.

The 4 exclusions

All scenario-level, all in gemma-faceted-search, each with its own reason,
and each verified not to leak — the sibling scenario sharing a requirement with
the code-path-identity exclusion is still counted as uncovered, so
.github#356 does not apply. They are the cases where no observation could
decide the claim: a PHP array key between two services (the endpoint's _meta
echoes no query), paging vs. scanning (byte-identical response, signal is a log
line), code-path identity, and l10n key names.

Where a requirement is merely broken or absent, an issue was filed instead.

Dead code: GemmaViewIndex.vue

Not concluded from "no references found" — that is an absence claim. Proven
positively: the bundle built with the file is byte-identical to the
bundle built without it (md5 b6cc3b81…, 5,215,895 bytes), so webpack cannot
reach it from any entry.

Positive control in the same bundle: OrganisationMergePanel 9 hits,
"Preview merge" 3, SuitesIndexView 6 — the search does find live
components. GemmaViewIndex: 0, and its unique string
"No GEMMA views are available": 0.

Cross-checked against all six ways a component can be reached here —
registry.js, customComponents.js, every string-keyed
component/widgetKey/target/handler value across manifest.json +
manifest.d/*.json (13 names, enumerated in full), router.js, dynamic
imports, and the bundle.

src/store/modules/view.js is now its consumer-less sibling and is also absent
from the bundle — left in place deliberately, it fronts a backend
view-enrichment API and deserves its own decision.

Newman verdict

Integration Tests (Newman) reports skipped, and a skipped job is no
verdict — so I checked rather than accepting the comment. It should stay
off, and the recorded reason is accurate. The only
*.postman_collection.json under the configured path is titled *"OpenRegister

  • Magic Mapper CSV Import Test"* — it exercises OpenRegister, not
    softwarecatalog; its base_url is the dev container's hostname, the workflow
    overrides it with http://localhost:8080 without the
    /index.php/apps/openregister/api suffix (so every request would 404), and one
    request uploads a file from a /var/www/html/custom_apps/... path that does
    not exist on the runner.

Separately worth knowing: the app's own collection,
postman/softwarecatalogus-tests.json, can never run under this workflow at
all — the run step is a flat glob on a filename suffix it does not have, in
a directory that is not the configured path, while the validator counts
collections recursively. A repo can satisfy the validator and run zero
collections.

Not done

Nothing was weakened: no baselines, no skipped or deleted tests, no
mode: 'serial', no widened timeouts, no waitForLoadState('networkidle').

A first-run-wizard dismiss helper was written, found not to work when tested
against the enabled app, and reverted rather than shipped with a docblock
claiming a capability it did not have. Local rigs need
occ app:disable firstrunwizard; CI does not have it enabled.

Refs #481 #482 #483 #484 #485

⚠️ CORRECTIONS — read these, the earlier body was wrong in two places

1. RETRACTED: "this PR's gate-19 passes over an empty scope"

That was wrong, and CI proved it. The first push made this PR's own gate-19
report FAIL — 21, not a pass. gate-19 scopes by file: touching
openspec/specs/gemma-faceted-search/spec.md to add four exclusions pulled
all 25 of that spec's scenarios into this PR's diff. A gate-19 remediation
PR is exactly the case where the "empty scope" intuition fails, because closing
findings means editing spec files.

Resolution: the four exclusions were removed from this changeset. Two of
that spec's 25 scenarios are unimplemented (#483), and an @e2e exclude asserts
"a browser cannot observe this" — a sentence that is false for both. So no
honest version of this PR could be green while editing that file. The exclusions
land with the #483 fixes instead.

With no spec file touched, gate-19 now SKIPs as na — "the diff touched NO
spec file". That is not a pass and is not claimed as one. The number that
means anything is the development → beta scope.

2. Updated numbers
development → beta scope 137 → 111
closed 26 scenarios, 19 test cases, 0 exclusions
this PR's own scope gate-19 na (skip) — no verdict

Negative control re-run on the final tree, perturbing a metric the gate counts:

step count
with gemma-faceted-search.spec.ts 111
removed 116 (+5 = its exactly-5 anchors)
restored 111

3. Two traps hit while proving the tests can fail

Both first read as "my tests are blind" — the wrong conclusion was one step away.

  • opcache.revalidate_freq = 60 on the nextcloud:34-apache image. A PHP
    plant is invisible for up to a minute; the first planted run passed 5/5 over
    genuinely broken code    . Verify the plant is LIVE before running the assay.
  • The planted constant was one of three copies of the GEMMA dimension list
    (FacetService::DIMENSIONS, a bare literal in FacetController::parseFilters(),
    and the per-object map literal). The plant was real and changed nothing on the
    response path.

4. gate-16 re-measured with an explicit base (.github#361)

check_spec_coverage.py:508 defaults HYDRA_GATE_BASE_REF to
origin/development, so a full-scope run on development diffs against nothing
and prints PASS. Re-measured here with the base named:

base files the gate inspected findings
origin/development 4 0
origin/beta 661 (183 PHP, 198 frontend) 0

softwarecatalog's gate-16 is honestly clean — the 0 is over a large real scope,
not over an empty one. #361 is real in the package but does not flatter this
app.

@rubenvdlinde
test(e2e): 21 real Playwright tests + 4 exclusions close gate-19 137 …
53b14de 
…-> 112

gate-19 (e2e-coverage) was softwarecatalog's only failing gate. Baseline
reproduced locally byte-identical to CI — 137 — against the CANONICAL gate
package (freshly cloned .github@main), scoped HYDRA_GATE_BASE_REF=origin/beta,
which is what CI resolves for the open development -> beta PR #197.

  start 137  ->  end 112   (21 tests + 4 exclusions)

Negative control, run on the final tree:
  with suite-wizard.spec.ts     112
  file removed                  118   (+6 = its exactly-6 @e2e anchors)
  file restored                 112

Every one of the 14 new tests was proven able to FAIL by a planted true
positive, each plant chosen so only its own test went red — e.g. `applicaties:
[]` in buildSuitePayload reddened only the submit test; renaming the third
wizard step reddened only the step-labels test. Two needed the plant to remove
TWO layers: Nextcloud core's SecurityMiddleware rejects an unauthenticated
request before the controller body runs, so the anon-POST test only reddened
once #[PublicPage] + #[NoCSRFRequired] were added as well.

Full suite on an isolated single-owner rig: 90 passed, 4 skipped, 0 failed.
The 4 skipped are pre-existing test.fixme cases, untouched.

WHAT THE TESTS FOUND (all filed, none worked around)

* #481 — every write through src/utils/adminApi.js failed CSRF. The helper
  sent no Nextcloud requesttoken, and none of its target controllers is
  #[NoCSRFRequired], so SEVEN shipped actions were unreachable from the UI:
  review submit, moderation approve/reject, federation peer add/remove/pull,
  EOL config save and "Sync now". GET is exempt, so every settings section
  rendered fine and only its buttons were dead. Fixed here, because the
  coverage could not exist without it: the same unchanged test goes green once
  the token is sent. src/store/modules/facets.js already used @nextcloud/axios
  and was never affected.

* #482 — the suite wizard's success result is never rendered. onSuiteCreated
  sets showWizard = false in the same synchronous tick setResult() runs, and
  the dialog is under v-if, so the result phase is destroyed before any render
  flush. Not a race. The test asserts the two THENs that hold and DISCLOSES the
  third in a comment block rather than quietly dropping it.

* #483 — two gemma-faceted-search requirements are unimplemented: empty facet
  dimensions are never disabled, and the facet cache is TTL-only with no
  invalidation path. Both are browser-observable, so both are left IN the
  count rather than excluded.

* #484 — sbom-import.spec.ts carries 16 file-level @e2e tags in a docblock that
  says, verbatim, "excluded from Playwright coverage below". gate-19 credits
  them (.github#343), so the honest debt is 153, not 137. Deliberately NOT
  "fixed" here — converting 16 positive claims into exclusions inside a PR that
  lowers the gate count is indistinguishable from gaming it.

* #485 — 23 whole-spec @e2e exclude markers retire 212 of 696 scenarios (30%),
  several on reasons that do not hold ("no live deployment in this pass";
  Vue modals excluded from browser testing; "is an interaction").

THE 4 EXCLUSIONS

All scenario-level, all in gemma-faceted-search, each with its own reason, and
each verified not to leak: the sibling scenario sharing a requirement with the
code-path-identity exclusion is still counted as uncovered (.github#356).
They are the cases where no observation could decide the claim — a PHP array
key between two services (the endpoint's _meta echoes no query), paging vs.
scanning (byte-identical response, signal is a log line), code-path IDENTITY
(two paths agreeing on every output are indistinguishable from one), and l10n
KEY names (the browser only ever sees resolved values).

Refused the exclusion where the requirement is merely broken or absent — those
are issues above, not waivers.

DEAD CODE

src/views/gemmaviews/GemmaViewIndex.vue deleted. Not concluded from "no
references found" — that is an absence claim. Proven positively: the bundle
built WITH the file present is BYTE-IDENTICAL to the bundle built without it
(md5 b6cc3b81..., 5,215,895 bytes), so webpack could not reach it from any
entry. Positive control in the same bundle: OrganisationMergePanel 9 hits,
"Preview merge" 3, SuitesIndexView 6 — the search does find live components.
GemmaViewIndex: 0, and its unique string "No GEMMA views are available": 0.
Cross-checked against the six ways a component can be reached in these apps —
registry.js, customComponents.js, every string-keyed component/widgetKey/
target/handler value in manifest.json + manifest.d/*.json (13 names, enumerated
in full), router.js, dynamic imports, and the bundle. The only surviving
mention was a proposal that had already flagged it as orphaned.

src/store/modules/view.js is now its only-consumer-less sibling and is also
absent from the bundle; left in place deliberately — it fronts a backend
view-enrichment API and deserves its own decision.

Refs #481 #482 #483 #484 #485
@github-actions

github-actions Bot commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

Quality Report — ConductionNL/softwarecatalog @ 5f27e79

Check PHP Vue Security License Tests
lint ✅
phpcs ✅
phpmd ✅
psalm ✅
phpstan ✅
phpmetrics ✅
eslint ✅
stylelint ✅
build ✅
check-manifest ✅
check-vue-demi ✅
test-l10n ✅
composer ✅ ✅ 128/128
npm ✅ ✅ 718/718
PHPUnit ✅
Newman ⏭️
Playwright ✅
Hydra gates ❌

Quality workflow — 2026-08-11 16:42 UTC

Download the full PDF report from the workflow artifacts.

@rubenvdlinde
test(e2e): 5 facet-contract tests; defer the 4 exclusions so this PR …
c06bb09 
…stays honest

Follow-up to the previous commit on this branch. Two changes:

1. ADDS tests/e2e/spec-coverage/gemma-faceted-search.spec.ts — 5 tests covering
   the facet CONTRACT and the panel's presence:
     - all four GEMMA dimensions present, empty ones as [] not omitted
     - an unsupported schema is rejected 400, naming the supported set
     - a text query narrows the aggregated set
     - a repeated identical request is served from cache; a different one is not
     - the GEMMA panel renders beside the still-working free-text search box

2. REVERTS the four @e2e exclude markers added to
   openspec/specs/gemma-faceted-search/spec.md in the previous commit.

WHY THE EXCLUSIONS ARE BEING DEFERRED

Editing that spec file pulls ALL 25 of its scenarios into this PR's diff scope
(gate-19 scopes by FILE, not by hunk). CI proved it: the previous commit made
this PR's own gate-19 report FAIL - 21, reproduced locally byte-identical.

Two of those 25 are unimplemented outright (#483: empty facet dimensions are
never disabled; the facet cache is TTL-only with no invalidation path). Both are
browser-observable, so neither can honestly carry an `@e2e exclude` — an
exclusion asserts "a browser cannot observe this", and for these two that
sentence is false. And no passing test can be written against behaviour that
does not exist.

So no honest version of this PR can be green while it edits that file. The four
exclusions — which are good, scenario-level, and individually reasoned — land
together with the #483 fixes instead. Cost: 4 scenarios of headline number.
Benefit: the gate keeps meaning what it says.

With the spec file untouched, gate-19 for this PR SKIPs as `na` ("the diff
touched NO spec file"). That is not a pass and is not claimed as one; the
honest figure remains the development -> beta scope.

  beta scope: 137 -> 111   (26 scenarios, 19 test cases, 0 exclusions)

WHAT THE FACET FIXTURES CANNOT DO, AND WHY THAT IS RECORDED

The facet VALUE scenarios (per-dimension counts, OR-within / AND-across,
URL round-tripping of a selection, saved views) are NOT claimed. POSTing a
module with {"standaardVersies":["x"],"referentieComponenten":["y"]} returns 200
with BOTH arrays silently emptied — they are relation fields and OpenRegister
drops bare strings without erroring. Seeding them needs real `element` objects
plus `relation` rows. A facet test seeded with data the product cannot produce
would prove nothing, so those scenarios stay uncovered and counted.

`no-text-query-returns-facets-over-the-full-rbac-scoped-set` is also left
uncovered on purpose: it needs the unparameterised aggregate, which is the exact
entry the cache never invalidates. Measured — findAll saw 3 modules while the
endpoint answered {"cached":true,"totalMatched":1}. The three available ways to
make it green (wait out the 1800s TTL, assert >= instead of ==, or add a
cache-busting param and test a different scenario) were all rejected.

Refs #483
@github-actions

github-actions Bot commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

Quality Report — ConductionNL/softwarecatalog @ d3a1649

Check PHP Vue Security License Tests
lint ✅
phpcs ✅
phpmd ✅
psalm ✅
phpstan ✅
phpmetrics ✅
eslint ✅
stylelint ✅
build ✅
check-manifest ✅
check-vue-demi ✅
test-l10n ✅
composer ✅ ✅ 128/128
npm ✅ ✅ 718/718
PHPUnit ✅
Newman ⏭️
Playwright ❌
Hydra gates ✅

Quality workflow — 2026-08-11 17:40 UTC

Download the full PDF report from the workflow artifacts.

@rubenvdlinde
fix(e2e): drop the facet-cache test — the CI instance configures no c…
d35596c 
…ache backend

CI caught this, not the dev rig: the test passed locally and failed on the
runner, on the first attempt and its retry.

The facet cache is `ICacheFactory::createDistributed(...)`, which degrades to a
NULL cache when Nextcloud has no memcache backend configured — nothing is
stored, so `_meta.cached` is always false and "a repeated identical request is
served from cache" has no observable behaviour there.

Measured on both sides rather than assumed:
  dev rig  `occ config:system:get memcache.local` -> \OC\Memcache\APCu, APCu
           present; the flag flips false -> true reliably.
  CI       the shared workflow configures no memcache at all (zero mentions of
           memcache/apcu in the job log); both calls returned `cached: false`.

The two tempting repairs were rejected. A `test.skip` guard on "is caching
available" can never be false on CI, so it would credit coverage that never
executes — the exact shape flagged fleet-wide. Asserting only that `cached` is
a boolean is an assertion that cannot fail.

So the scenario is left uncovered and COUNTED, with the reasoning recorded at
the point where the test used to be. It becomes coverable when the CI instance
configures a cache backend.

  beta scope: 137 -> 112   (25 scenarios, 18 test cases, 0 exclusions)

Negative control on the final tree:
  with tests/e2e/spec-coverage/gemma-faceted-search.spec.ts   112
  file removed                                                116  (+4 = its 4 anchors)
  file restored                                               112

Refs #483
@github-actions

github-actions Bot commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

Quality Report — ConductionNL/softwarecatalog @ 96761f4

Check PHP Vue Security License Tests
lint ✅
phpcs ✅
phpmd ✅
psalm ✅
phpstan ✅
phpmetrics ✅
eslint ✅
stylelint ✅
build ✅
check-manifest ✅
check-vue-demi ✅
test-l10n ✅
composer ✅ ✅ 128/128
npm ✅ ✅ 718/718
PHPUnit ✅
Newman ⏭️
Playwright ✅
Hydra gates ✅

Quality workflow — 2026-08-11 18:02 UTC

Download the full PDF report from the workflow artifacts.

@rubenvdlinde
rubenvdlinde merged commit ebc93ff into development Aug 11, 2026
29 checks passed
@rubenvdlinde
rubenvdlinde deleted the fix/gate19-e2e-coverage branch August 11, 2026 18:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@rubenvdlinde