test: Playwright visual-regression harness (v1.1.1 baselines)#60
Merged
Conversation
Models quantecon-book-theme's visual-regression setup, adapted for this theme's runtime nature: a small self-contained MyST fixture (intro/features/notebook — the notebook bakes in stream/execute_result/error outputs to exercise the @myst-theme v1.0.0 output-node AST change) is rendered via a live `myst start`, with the theme under test chosen at runtime via the THEME_TEMPLATE env var (a local build dir, or a GitHub archive zip). - playwright.config.ts (desktop + mobile-chrome), tests/visual/theme.spec.ts (full-page snapshots), serve.sh, README documenting the before/after flow. - Baselines captured from the deployed v1.1.1 theme (the last release), so we can diff the 2.0.0 upgrade against them. - npm scripts test:visual / test:visual:update; @playwright/test devDep. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Adds a Playwright-based visual regression harness for this theme’s runtime rendering (via myst start) using a self-contained MyST fixture project and committed snapshot baselines (intended to represent v1.1.1) to support reviewing/accepting visual changes from the upcoming upgrade work.
Changes:
- Introduces a visual fixture project (
tests/visual/fixture/) including markdown surfaces and a notebook with baked outputs. - Adds Playwright configuration + spec to run full-page screenshot comparisons across desktop and mobile.
- Wires up local serving (
serve.sh), documentation, npm scripts, and Playwright devDependencies; ignores generated artifacts.
Reviewed changes
Copilot reviewed 9 out of 17 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
tests/visual/theme.spec.ts |
Adds full-page screenshot tests for intro/features/notebook routes. |
tests/visual/serve.sh |
Starts a myst start server for the fixture with a theme selected via THEME_TEMPLATE. |
tests/visual/README.md |
Documents how to run and update visual baselines locally. |
tests/visual/fixture/notebook.ipynb |
Adds notebook surface to exercise output rendering (stream/execute_result/error). |
tests/visual/fixture/myst.yml.in |
Provides MyST config template with __THEME__ placeholder for runtime injection. |
tests/visual/fixture/intro.md |
Adds landing-page content surface for baseline snapshots. |
tests/visual/fixture/features.md |
Adds rich content surface (math/code/admonitions/tables/definitions) for baseline snapshots. |
playwright.config.ts |
Adds Playwright runner config, devices, snapshot pathing, and webServer integration. |
package.json |
Adds test:visual scripts and @playwright/test devDependency. |
package-lock.json |
Locks Playwright-related dependencies and updates lock metadata accordingly. |
.gitignore |
Ignores generated Playwright and MyST build artifacts while keeping snapshots committed. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
`waitForLoadState("networkidle")` never fires against the runtime theme
(it can hold a persistent connection open), so every snapshot test timed
out before screenshotting. Wait for `load` + fonts.ready + a short beat
instead. (Surfaced while validating the BannerStateProvider fix, where the
1.3.0 app kept a connection open.)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Addresses Copilot review on #60: - serve.sh default PORT 3000 -> 3111 to match playwright.config.ts (tests failed by default when PORT was unset: config waited on 3111, server on 3000). - Escape sed replacement specials (\\ & |) so a THEME_TEMPLATE URL containing & (query params) no longer corrupts myst.yml. - README: serve.sh runs `myst start` (not `--headless`, which was removed). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
mmcky
added a commit
that referenced
this pull request
Jun 4, 2026
…rendering) (#61) * fix: wrap page tree in BannerStateProvider (@myst-theme 1.x) @myst-theme 1.x moved banner visibility/height into a React context (`BannerStateProvider` in @myst-theme/providers); a child component calls `useBannerState()` during SSR. Our custom app/components/Page.tsx provider stack didn't include the provider, so every page threw "useBannerState must be used from within BannerStateProvider" and returned HTTP 500. `compile` + `prod:build` pass (it's a runtime/SSR error), and it had never surfaced because the deployed bundle is still v1.1.1 (@myst-theme 0.14) — the 1.x upgrade had never actually rendered. Wrap the tree in BannerStateProvider, mirroring upstream's page route. Found + validated with the new visual-regression harness (#60): all pages go 500 -> 200, notebook stream/error outputs render, and the diff vs the v1.1.1 baselines is a modest 2-4% (expected styling from the 0.14->1.3.0 upgrade). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: move BannerStateProvider into the shared NavigationAndArticleWrapper Addresses Copilot review: the provider was only on the happy path (Page), but ErrorBoundary -> ErrorPage also renders NavigationAndArticleWrapper, whose useSidebarHeight -> useBannerState would still 500 the error UI. Place BannerStateProvider inside NavigationAndArticleWrapper (used by both Page and ErrorPage) and drop the now-redundant wrap in Page.tsx, so every tree that renders the wrapper has the provider. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
mmcky
added a commit
that referenced
this pull request
Jun 4, 2026
Bump 1.1.1 -> 2.0.0 across package.json, package-lock.json and template.yml, and finalize the CHANGELOG [2.0.0] section. 2.0.0 is the @myst-theme 0.14 -> 1.x upgrade (breaking: new notebook output-node AST; Node engine >=20) plus the technical-review pass, the visual-regression harness (#60), and three pre-release fixes for regressions the 1.x upgrade introduced and caught before deploy: the BannerStateProvider 500 (#61), the full-bleed content column (#62), and the Remix 1.19 hydration reload loop (#63). CHANGELOG corrections vs the old [Unreleased] draft: drop the reverted #29 Remix 1.17->1.19 bump (2.0.0 stays on the v1.0.1 ~1.17.0 pin, so no net Remix change), correct the Node floor to >=20, and restate the Security note now that the older Remix v1 toolchain remains. Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a Playwright visual-regression harness — modelled on
quantecon-book-theme's setup but adapted for this theme's runtime nature (rendered through a livemyst start, not static HTML).What's here
tests/visual/fixture/):intro.md,features.md(math/code/admonitions/tables/figures), andnotebook.ipynbwith baked stream / execute_result / error outputs — the surface the@myst-themev1.0.0 output-node AST change touches.serve.shrunsmyst startwith the theme chosen at runtime viaTHEME_TEMPLATE(a local build dir or a GitHub archive zip).playwright.config.ts(desktop + mobile-chrome),theme.spec.ts(full-page snapshots), README.test:visual/test:visual:update;@playwright/testdevDep.Baselines = the last release (v1.1.1)
The committed
__snapshots__/were captured against the deployed v1.1.1 theme (…/quantecon-theme/archive/refs/heads/main.zip) — validated rendering live, including the notebook stream/error outputs. So we can diff the 2.0.0 upgrade against the last release:Once the 2.0.0 visual changes are reviewed/accepted, refresh baselines with
test:visual:update.Notes
.nvmrc),mystCLI, andnpx playwright install chromium.fixture/myst.yml,fixture/_build/,playwright-report/are gitignored.visual.ymlmatrix that builds the 2.0.0 bundle and runs the snapshots).🤖 Generated with Claude Code