Skip to content

test: Playwright visual-regression harness (v1.1.1 baselines)#60

Merged
mmcky merged 3 commits into
mainfrom
test/visual-regression-harness
Jun 4, 2026
Merged

test: Playwright visual-regression harness (v1.1.1 baselines)#60
mmcky merged 3 commits into
mainfrom
test/visual-regression-harness

Conversation

@mmcky
Copy link
Copy Markdown
Contributor

@mmcky mmcky commented Jun 4, 2026

Adds a Playwright visual-regression harness — modelled on quantecon-book-theme's setup but adapted for this theme's runtime nature (rendered through a live myst start, not static HTML).

What's here

  • Self-contained fixture (tests/visual/fixture/): intro.md, features.md (math/code/admonitions/tables/figures), and notebook.ipynb with baked stream / execute_result / error outputs — the surface the @myst-theme v1.0.0 output-node AST change touches.
  • serve.sh runs myst start with the theme chosen at runtime via THEME_TEMPLATE (a local build dir or a GitHub archive zip).
  • playwright.config.ts (desktop + mobile-chrome), theme.spec.ts (full-page snapshots), README.
  • npm scripts test:visual / test:visual:update; @playwright/test devDep.

Baselines = the last release (v1.1.1)

The committed __snapshots__/ were captured against the deployed v1.1.1 theme (…/quantecon-theme/archive/refs/heads/main.zip) — validated rendering live, including the notebook stream/error outputs. So we can diff the 2.0.0 upgrade against the last release:

make build-theme
THEME_TEMPLATE="$PWD/.deploy/quantecon-theme" npm run test:visual   # diff vs v1.1.1

Once the 2.0.0 visual changes are reviewed/accepted, refresh baselines with test:visual:update.

Notes

  • Requires Node 24 (.nvmrc), myst CLI, and npx playwright install chromium.
  • Generated fixture/myst.yml, fixture/_build/, playwright-report/ are gitignored.
  • Follow-up: wire this into CI (a visual.yml matrix that builds the 2.0.0 bundle and runs the snapshots).

🤖 Generated with Claude Code

Models quantecon-book-theme's visual-regression setup, adapted for this
theme's runtime nature: a small self-contained MyST fixture
(intro/features/notebook — the notebook bakes in stream/execute_result/error
outputs to exercise the @myst-theme v1.0.0 output-node AST change) is rendered
via a live `myst start`, with the theme under test chosen at runtime via the
THEME_TEMPLATE env var (a local build dir, or a GitHub archive zip).

- playwright.config.ts (desktop + mobile-chrome), tests/visual/theme.spec.ts
  (full-page snapshots), serve.sh, README documenting the before/after flow.
- Baselines captured from the deployed v1.1.1 theme (the last release), so we
  can diff the 2.0.0 upgrade against them.
- npm scripts test:visual / test:visual:update; @playwright/test devDep.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Copilot AI review requested due to automatic review settings June 4, 2026 04:38
Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds a Playwright-based visual regression harness for this theme’s runtime rendering (via myst start) using a self-contained MyST fixture project and committed snapshot baselines (intended to represent v1.1.1) to support reviewing/accepting visual changes from the upcoming upgrade work.

Changes:

  • Introduces a visual fixture project (tests/visual/fixture/) including markdown surfaces and a notebook with baked outputs.
  • Adds Playwright configuration + spec to run full-page screenshot comparisons across desktop and mobile.
  • Wires up local serving (serve.sh), documentation, npm scripts, and Playwright devDependencies; ignores generated artifacts.

Reviewed changes

Copilot reviewed 9 out of 17 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
tests/visual/theme.spec.ts Adds full-page screenshot tests for intro/features/notebook routes.
tests/visual/serve.sh Starts a myst start server for the fixture with a theme selected via THEME_TEMPLATE.
tests/visual/README.md Documents how to run and update visual baselines locally.
tests/visual/fixture/notebook.ipynb Adds notebook surface to exercise output rendering (stream/execute_result/error).
tests/visual/fixture/myst.yml.in Provides MyST config template with __THEME__ placeholder for runtime injection.
tests/visual/fixture/intro.md Adds landing-page content surface for baseline snapshots.
tests/visual/fixture/features.md Adds rich content surface (math/code/admonitions/tables/definitions) for baseline snapshots.
playwright.config.ts Adds Playwright runner config, devices, snapshot pathing, and webServer integration.
package.json Adds test:visual scripts and @playwright/test devDependency.
package-lock.json Locks Playwright-related dependencies and updates lock metadata accordingly.
.gitignore Ignores generated Playwright and MyST build artifacts while keeping snapshots committed.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread playwright.config.ts
Comment thread tests/visual/serve.sh Outdated
Comment thread tests/visual/README.md Outdated
mmcky and others added 2 commits June 4, 2026 15:20
`waitForLoadState("networkidle")` never fires against the runtime theme
(it can hold a persistent connection open), so every snapshot test timed
out before screenshotting. Wait for `load` + fonts.ready + a short beat
instead. (Surfaced while validating the BannerStateProvider fix, where the
1.3.0 app kept a connection open.)

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Addresses Copilot review on #60:
- serve.sh default PORT 3000 -> 3111 to match playwright.config.ts (tests
  failed by default when PORT was unset: config waited on 3111, server on 3000).
- Escape sed replacement specials (\\ & |) so a THEME_TEMPLATE URL containing
  & (query params) no longer corrupts myst.yml.
- README: serve.sh runs `myst start` (not `--headless`, which was removed).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@mmcky mmcky merged commit 5fd1c65 into main Jun 4, 2026
1 check passed
@mmcky mmcky deleted the test/visual-regression-harness branch June 4, 2026 05:53
mmcky added a commit that referenced this pull request Jun 4, 2026
…rendering) (#61)

* fix: wrap page tree in BannerStateProvider (@myst-theme 1.x)

@myst-theme 1.x moved banner visibility/height into a React context
(`BannerStateProvider` in @myst-theme/providers); a child component calls
`useBannerState()` during SSR. Our custom app/components/Page.tsx provider
stack didn't include the provider, so every page threw
"useBannerState must be used from within BannerStateProvider" and returned
HTTP 500. `compile` + `prod:build` pass (it's a runtime/SSR error), and it had
never surfaced because the deployed bundle is still v1.1.1 (@myst-theme 0.14) —
the 1.x upgrade had never actually rendered.

Wrap the tree in BannerStateProvider, mirroring upstream's page route.

Found + validated with the new visual-regression harness (#60): all pages go
500 -> 200, notebook stream/error outputs render, and the diff vs the v1.1.1
baselines is a modest 2-4% (expected styling from the 0.14->1.3.0 upgrade).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix: move BannerStateProvider into the shared NavigationAndArticleWrapper

Addresses Copilot review: the provider was only on the happy path (Page),
but ErrorBoundary -> ErrorPage also renders NavigationAndArticleWrapper,
whose useSidebarHeight -> useBannerState would still 500 the error UI.

Place BannerStateProvider inside NavigationAndArticleWrapper (used by both
Page and ErrorPage) and drop the now-redundant wrap in Page.tsx, so every
tree that renders the wrapper has the provider.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
mmcky added a commit that referenced this pull request Jun 4, 2026
Bump 1.1.1 -> 2.0.0 across package.json, package-lock.json and
template.yml, and finalize the CHANGELOG [2.0.0] section.

2.0.0 is the @myst-theme 0.14 -> 1.x upgrade (breaking: new notebook
output-node AST; Node engine >=20) plus the technical-review pass, the
visual-regression harness (#60), and three pre-release fixes for
regressions the 1.x upgrade introduced and caught before deploy: the
BannerStateProvider 500 (#61), the full-bleed content column (#62), and
the Remix 1.19 hydration reload loop (#63).

CHANGELOG corrections vs the old [Unreleased] draft: drop the reverted
#29 Remix 1.17->1.19 bump (2.0.0 stays on the v1.0.1 ~1.17.0 pin, so no
net Remix change), correct the Node floor to >=20, and restate the
Security note now that the older Remix v1 toolchain remains.

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants