Migrate web E2E to Playwright with approval-gated CI#365
Merged
Conversation
Contributor
CI Coverage & Test Summary
Suites: 62 passed, 0 failed (62 total) · Tests: 658 passed, 0 failed (658 total) ✅ All reported test suites passed. Coverage artifacts: Updated at: May 23, 2026 at 10:32 PM PDT |
Contributor
Last deployed: 22:08 PDT |
Contributor
DB & Integration Test Summary
Full logs: Updated at: May 23, 2026 at 10:34 PM PDT |
There was a problem hiding this comment.
Pull request overview
This PR migrates centralized web E2E testing from Maestro (beta web support) to a Playwright suite under tests/e2e/web/, and adds an approval-gated GitHub Actions workflow to run those E2Es against the PR preview deployment after an explicit approver review.
Changes:
- Added Playwright web E2E suite (fixtures, global setup/teardown, config) and replaced the previous Maestro web flow.
- Introduced an approval-gated CI workflow (
e2e-web-pr-approval.yml) that validates the preview deployment SHA, optionally bootstraps CloudFront signed cookies, runs Playwright, uploads artifacts, and posts a formatted PR comment. - Added supporting scripts (signed-cookie bootstrap URL generator + PR comment formatter + unit tests), updated docs, and bumped dependencies for Playwright.
Reviewed changes
Copilot reviewed 32 out of 34 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
tests/e2e/web/specs/profile/profile-edit.spec.ts |
Playwright spec for editing profile display name. |
tests/e2e/web/specs/navigation/user-menu.spec.ts |
Playwright spec validating authenticated user-menu navigation. |
tests/e2e/web/specs/marketing/policies.spec.ts |
Playwright spec covering policy pages rendering. |
tests/e2e/web/specs/marketing/landing.spec.ts |
Playwright specs for marketing landing + CTA navigation. |
tests/e2e/web/specs/billing/usage.spec.ts |
Playwright spec validating billing usage view. |
tests/e2e/web/specs/billing/plans.spec.ts |
Playwright spec validating billing plans view. |
tests/e2e/web/specs/billing/overview.spec.ts |
Playwright spec validating billing overview hub. |
tests/e2e/web/specs/billing/navigation.spec.ts |
Playwright spec validating billing tab navigation. |
tests/e2e/web/specs/auth/signup.spec.ts |
Playwright spec for signup happy-path/confirm-email branch. |
tests/e2e/web/specs/auth/signup-invite.spec.ts |
Playwright spec for invalid invite handling. |
tests/e2e/web/specs/auth/protected-routes.spec.ts |
Playwright specs ensuring unauthenticated redirects work. |
tests/e2e/web/specs/auth/logout.spec.ts |
Playwright spec for logout + re-login flow. |
tests/e2e/web/specs/auth/login.spec.ts |
Playwright spec for seeded-user login. |
tests/e2e/web/specs/auth/login-validation.spec.ts |
Playwright specs for login form validation + signup link. |
tests/e2e/web/specs/auth/forgot-password.spec.ts |
Playwright specs for forgot-password UX variants. |
tests/e2e/web/specs/admin/access.spec.ts |
Playwright specs for admin access + navigation. |
tests/e2e/web/playwright.config.ts |
Playwright runner configuration (reporters, webServer behavior, retries/workers). |
tests/e2e/web/global-teardown.ts |
Teardown that deletes seeded users via service-role client. |
tests/e2e/web/global-setup.ts |
Seeds standard/admin users and saves storageState for authenticated fixtures. |
tests/e2e/web/fixtures/auth.fixture.ts |
Shared Playwright fixtures/helpers for authenticated/admin pages + form helpers. |
tests/e2e/web/env.ts |
Centralized env helpers for base URL/path handling + preview Supabase env mapping. |
tests/e2e/web/flows/home.yaml |
Removes old Maestro web YAML flow. |
scripts/run-e2e.sh |
Switches web E2E execution from Maestro to Playwright. |
scripts/pr-preview/generate-preview-bootstrap-url.sh |
New helper to generate signed-cookie bootstrap URL for preview access. |
scripts/ci/format-playwright-e2e-pr-comment.mjs |
New formatter to post Playwright JSON results as a PR comment. |
scripts/__tests__/format-playwright-e2e-pr-comment.test.mjs |
Adds unit tests for the PR comment formatter script. |
scripts/__tests__/fixtures/playwright-e2e-report.json |
Fixture JSON report used by the formatter unit tests. |
package.json |
Replaces web E2E script with Playwright commands; adds @playwright/test. |
package-lock.json |
Locks Playwright dependencies (@playwright/test, playwright, playwright-core). |
docs/TESTING.md |
Updates testing documentation to reflect Playwright web E2E + approval-gated CI. |
docs/ARCHITECTURE.md |
Documents the approval-gated Playwright web E2E workflow. |
.gitignore |
Ignores Playwright E2E artifacts and auth state files under tests/e2e/web/. |
.github/workflows/pr-preview-environment.yml |
Refactors signed-cookie bootstrap URL generation to the new script. |
.github/workflows/e2e-web-pr-approval.yml |
Adds the new approval-gated Playwright web E2E workflow. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
ZappoMan
force-pushed
the
tests/migrate-web-e2e-playwright
branch
from
255d223 to
0964e61
Compare
Web flows now run via Playwright locally and against PR previews when @ZappoMan approves; mobile Maestro is unchanged.
Addresses GHSA-7mvr-c777-76hp (npm audit high severity).
Parse Playwright JSON results to post structured pass/fail/retry counts and test breakdowns on approval runs.
…cies. Seed an admin user alongside the standard test user so critical flows like logout, billing tabs, and admin access are exercised end-to-end.
…s gate. The db-tests job was failing when migration-filenames was skipped on PRs without supabase schema changes, even though pgTAP and integration tests passed in supabase-tests.
…egistration. Pass WEB_URL as deploy origin with WEB_BASE_PATH for preview E2E, stop logging signed bootstrap URLs via eval, register the Playwright comment formatter test in test:unit:scripts, and clarify spec vs test-case counts in docs.
ZappoMan
force-pushed
the
tests/migrate-web-e2e-playwright
branch
from
0964e61 to
e22ea5f
Compare
…ignore. Cover blob reports and root-level Playwright cache paths so local test runs do not leave untracked temporary files.
Authors who cannot self-approve can start the Playwright preview workflow without @ZappoMan approval.
issue_comment workflows only run from main; add a dispatcher there plus label and workflow_dispatch triggers on the Playwright workflow.
3 tasks
Accept the latest successful pr-N-preview deployment when HEAD only changed CI/docs and no web-relevant files changed since that deploy.
ZappoMan
added
run-e2e
Accept /pr-N landing URLs with or without a trailing slash instead of requiring the URL to end in /.
Use a single HTML table with colspan file headers and a gray background band for easier scanning in PR comments.
…t trim. Grant issues:write for E2E PR comments, declare Playwright env vars for type-check, race signup outcomes, and match Run e2e comments case-insensitively after trim.
Reuse the summary section icons in each test's Status column for quicker scanning.
ZappoMan
added
run-e2e
Contributor
End 2 End - Test ResultsSummary
Test Environment🌐 Domain: https://deploy.beakerstack.com/pr-365/ Details
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Poll for slower preview responses, surface UI errors, and force open signup mode during global setup.
ZappoMan
added
run-e2e
Bump email rate limits and enable autoconfirm during Playwright setup so signup and password-reset specs are not blocked by shared preview SMTP quotas.
ZappoMan
added
run-e2e
Disable Playwright webServer for all preview targets, snapshot/restore waitlist signup mode and patched auth fields, and standardize run-e2e.sh preview env wiring with --skip-web support.
4 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
tests/e2e/web/(16 spec files, 26 test cases covering auth, billing, admin, policies, navigation, profile, and marketing).github/workflows/e2e-web-pr-approval.yml) that runs against the PR preview after @ZappoMan approves, verifies preview SHA, bootstraps signed cookies, and posts formatted results to the PRTest plan
npm run test:e2e:web:install(first run / after Playwright upgrade)supabase start+npm run test:e2e:weblocally — all 26 test cases passdevelop, get preview deploy, approve as @ZappoMan — E2E workflow runs and posts comment with summary table + per-spec detailstests/e2e/mobile/)