e2e: visual-regression project excluded from the E2E gate until its baselines are regenerated in CI

[rubenvdlinde]

Context

#17 turns on the shared E2E Tests (Playwright) job. Of the 12 failures on that PR, 10 were fixed at their cause. 2 are excluded from the gate and tracked here.

This follows the fleet precedent set by openregister — "a gate that is red on arrival is a gate nobody turns on" — enable a subset that genuinely passes, wire it as the gate, and grow it. This issue is the visible half of that trade: the excluded work, named.

Excluded specs (2 failures)

Both in tests/e2e/visual/petstore.visual.spec.ts, project visual:

spec	test	failure
tests/e2e/visual/petstore.visual.spec.ts:19	Petstore — visual baselines › dashboard	toHaveScreenshot pixel diff vs dashboard-visual-linux.png
tests/e2e/visual/petstore.visual.spec.ts:23	Petstore — visual baselines › examples list	538684 pixels (ratio 0.53 of all image pixels) are different vs examples-visual-linux.png

Why excluded rather than fixed

The visual project's own config entry has always documented itself as opt-in and non-gating, with the reason spelled out:

PLATFORM CAVEAT: PNG baselines are host-font/GPU specific, so a CI Linux runner will not byte-match a dev-container baseline; the visual project must regenerate its baselines in-CI before it can gate.

That intent was never enforced. The shared workflow runs a bare npx playwright test --config=…, which runs every project, so visual gated the pipeline regardless of what its comment said. The committed baselines were shot in the NC 34 docker dev container; CI runs NC stable31 with different theming and different font rendering, so a ~53% diff is the expected outcome and says nothing about a petstore regression.

The project is now opt-in via PLAYWRIGHT_VISUAL=1 — which is what the comment already claimed was true.

Deliberately NOT done: re-baselining from a CI run. Committing baselines generated by the very run that would then be compared against them makes the test self-comparing — it would go green while proving nothing. Re-baselining has to be a considered act with the screenshots reviewed by a human.

To close this issue

1. Generate baselines on a CI runner (NC stable31, Linux) and review the PNGs before committing them.
2. Confirm the diff is stable across two independent runs — a baseline that only matches its own run is worthless.
3. Re-gate by removing the PLAYWRIGHT_VISUAL guard in playwright.config.ts.
4. Verify with a truncation control: truncate (never delete — ensureBundleBuilt() rebuilds a deleted bundle) the JS bundle and confirm the visual tests fail. A missing bundle returns HTTP 200 text/html, never 404, so a suite that still passes with an empty bundle is not testing the app.

Running them meanwhile

PLAYWRIGHT_VISUAL=1 npx playwright test --project visual
PLAYWRIGHT_VISUAL=1 npx playwright test --project visual --update-snapshots

No test was skipped, no assertion weakened, no timeout raised and no error allow-list widened.