Skip to content

Visual-regression snapshots that are the docs screenshots - #166

Merged
Exelord merged 1 commit into
mainfrom
claude/bold-cannon-hmsma2
Jul 26, 2026
Merged

Visual-regression snapshots that are the docs screenshots#166
Exelord merged 1 commit into
mainfrom
claude/bold-cannon-hmsma2

Conversation

@Exelord

@Exelord Exelord commented Jul 26, 2026

Copy link
Copy Markdown
Member

One pipeline, two jobs. packages/cloud/tests/visual.test.ts drives the real dashboard (built SPA + real platform on ephemeral Postgres + fake S3, seeded through the real /v1/ingest wire) in a real Chromium across the 9 documented surfaces, and compares each capture against its committed baseline — where the baseline IS the image the docs site embeds (apps/docs/src/assets/screenshots/*.png, the exact files the cloud docs already reference).

So a UI change either fails here as a visual regression, or is accepted with VX_UPDATE_SNAPSHOTS=1, which rewrites the baselines and therefore the docs screenshots in the same commit. Docs screenshots can no longer silently rot behind the product.

Determinism by construction

  • Seed anchored to a fixed epoch — 18 days of history + 5 same-day runs + a featured run with staggered per-task wallclock (so the flamegraph shows real parallelism), shaped so every documented card has something true to show (flaky task, a task that got slower, cache hits, a red run on a feature branch).
  • Browser clock frozen to that instant via addInitScript, so "2h ago" renders identically forever.
  • Animations/transitions disabled before the shutter, and the shutter fires only when two consecutive captures are byte-identical (stableShot — a measurement, not a magic timeout).
  • Comparison is dependency-free: helpers/png.ts hand-rolls the PNG reader (IHDR/IDAT + node:zlib inflate + all five unfilters incl. Paeth → RGBA) and a per-channel-tolerance differ, following the tar.ts / sigv4.ts precedent. ~100 ms per 3200×2000 image.

Two real defects surfaced by building it

  1. box-sizing was never reset. The UnoCSS preflight lacked the universal border-box rule, so every padded full-width element overflowed by exactly its padding — scrollWidth 1648 vs clientWidth 1600 on every dashboard page (a permanent horizontal scrollbar, and why the old screenshots clipped their right-hand column). Fixed; measured 1600/1600 after.
  2. The committed perf guard was silently skipping. bun test doesn't consult NODE_PATH and this container's Playwright is a global install, so ui-perf.test.ts had never resolved a browser. A shared helpers/playwright.ts loader (env override → NODE_PATH → conventional global prefixes, importing the package directory to bypass node resolution) now serves both suites — the perf guard runs for the first time (5 pass).

Verification

  • Differential: perturbing shell padding p-6p-10 reds 5 shots at 8–11% of pixels, with each capture parked in tmp for eyeballing and an actionable message; reverted → green.
  • Deterministic across repeated runs; 3 consecutive full cloud-suite runs at 549 pass / 0 fail.
  • Gates: oxfmt --check . clean · oxlint --type-aware --type-check 0 · cloud 549/0 · core from root 1286/0 · the docs site builds clean on the regenerated images.
  • Skips stay honest: no browser or no built SPA → skip, never fail (CI has neither today, so it skips there exactly like the perf guard). Baselines are environment-pinned — documented in the suite header and in a new README placed where someone would try to hand-replace an image.

No schema/wire/CACHE bump (test infra + one CSS reset rule).

🤖 Generated with Claude Code

https://claude.ai/code/session_01RW7aso5j5CrBo7cjyET23D


Generated by Claude Code

One pipeline, two jobs. tests/visual.test.ts drives the real dashboard
(built SPA + real platform on ephemeral Postgres, seeded through the
real ingest wire) in a real Chromium across the 9 documented surfaces
and compares each capture against its committed baseline — where the
baseline IS the image the docs site embeds. A UI change either fails
here as a visual regression, or is accepted with VX_UPDATE_SNAPSHOTS=1,
which rewrites the baselines and therefore the docs screenshots in the
same commit. Docs can no longer rot behind the product.

Determinism by construction: the seed is anchored to a fixed epoch, the
browser clock is frozen to that instant, animations are disabled, and
the shutter fires only when two consecutive captures are byte-identical
(a measurement, not a magic timeout). Comparison is dependency-free —
helpers/png.ts hand-rolls the PNG reader and a tolerance differ, the
tar.ts/sigv4.ts precedent.

Building it surfaced two real defects:

- box-sizing was never reset, so every padded full-width element
  overflowed by exactly its padding — scrollWidth 1648 vs clientWidth
  1600 on every dashboard page, a permanent horizontal scrollbar (and
  why the old screenshots clipped their right column). Fixed in the
  UnoCSS preflight; measured 1600/1600 after.
- The committed perf guard was silently skipping: bun test doesn't
  consult NODE_PATH and playwright here is a global install, so it had
  never resolved a browser. A shared helpers/playwright.ts loader now
  serves both suites and the perf guard runs for the first time.

Differentially verified: perturbing shell padding reds 5 shots at
8-11% of pixels with the capture parked for inspection; reverted, green.
Skips stay honest — no browser or no built SPA means skip, never fail.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RW7aso5j5CrBo7cjyET23D
@Exelord
Exelord merged commit 6dd19f2 into main Jul 26, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants