better playwright fixtures: cwd, edit, $

[pcattori]

I'm bringing over the testing utils I created for react-router-templates which are mostly Playwright fixtures.

The goal is for end-to-end tests to stick as close to real user setups and workflows as possible while making the pit of success for writing good (correct, thorough) tests as wide as possible.

import getPort from "get-port"

import { test } from "./helpers/fixtures"
import * as Stream from "./helpers/stream"

test("hello world", async ({ edit, $, page }) => {
  await $("pnpm typecheck")
  
  const port = await getPort()
  const url = `http://localhost:${port}`
  
  const dev = $(`pnpm dev --port ${port}`)
  await Stream.match(dev.stdout, url)
  
  await page.goto(url, { waitUntil: "networkidle" })

  await edit({
    "app/routes/_index.tsx": `
      export default function HelloWorld() {
        return <h1>Hello, world!</h1>
      }
    `,
    "vite.config.ts": (contents) => contents.replace(
      "import { reactRouter }",
      "import { unstable_reactRouterRSC as reactRouter }"
    ),
  })
  
  expect(page.locator("...")).toHaveText("...")
})

What do Playwright fixtures do?

1. Playwright statically analyzes which fixtures are used for each test
In this test, we use the built-in page fixture as well as our own custom edit and $ fixtures

2. Playwright determines dependency graph for fixtures
Our edit and $ fixtures depend on another one of our custom fixtures called cwd which is in charge of setting up a temporary directory for out test. What's cool about this is that edit and $ know about cwd, so I've set them up to automatically do everything relative to that path.

3. Playwright runs pre-test code for fixtures in dependency order
For us, this means that cwd pre-test actually runs now and creates the temporary directory. edit and $ don't do much in the pre-test phase. Notably, the fixture get executed separately for each test, so we get file system isolation for each test by default!

cwd also registers the temporary directory path as an attachment. That means any time a test fails that used these new fixtures, it will include the cwd path neatly in the test output so you can immediately dive into that temporary dir and do some debugging.

4. Playwright runs your test
We start by running pnpm typecheck just like a user would in their own terminal. If the command fails, we'll get an error thrown here with the exit code and error message. Then we asynchronously spin up a Vite dev server via pnpm dev using the --port flag with a unique port so we can run any other tests with Vite dev servers in parallel and without port conflicts.

We could create an abstraction for this like const { process, port } = viteDev() but I personally like how the pnpm dev command is right there in front of you.

Next, we wait until the dev process has a URL with our expected port in stdout via Stream.match (this API is in flight, up for bikeshedding)

Then, we make edits to our app code while the dev server is running. You can either provide a string or a (contents: string) => string function to manipulate the contents of each edited file.

At any point, you can use Playwright's built-in page fixture to test the page.

5. Playwright runs post-test code for fixtures in reverse dependency order
Now we give our fixtures a chance to clean up. edit doesn't have any clean up, but $ keep track of each process it spawned and gracefully terminates them via .kill() after the test is done to prevent any lingering, zombie processes from tests.

Currently, cwd does not clean up its own temporary directory because I am emulating how our tests worked previously, where we would only clean up integration/.tmp when all tests passed. Not sure if this was intentional or just a limitation of our previous flow, so I'm up to change that if we want.

Fixtures-options

By default, tests use the vite-6-template, but I've also created two fixtures-options that let you configure tests before they run: template and files.

test.use({
  template: "vite-rolldown-template",
  files: {
    "app/start-with-this-file.ts": `export const hello = "world"`
    // same shape you would use for the `edit` fixture
  }
})
test("stuff", () => {/* ... */ })

test.use configures all tests in its scope, so you can use test.describe to do multiple different setups in the same file:

test.describe("Vite Rolldown", () => {
  test.use({ template: "vite-rolldown-template" })
  test("stuff", () => {/* ... */ })
})

test.describe("Vite 7 Beta", () => {
  test.use({ template: "vite-7-beta-template" })
  test("stuff", () => {/* ... */ })
})

You can even do this programmatically with a loop:

const templates: Array<string> = [/* ... */]
for (const template of templates) {
  test.describe(template, () => {
    test.use({ template })
    test("stuff", () => {/* ... */})
  })
}

[changeset-bot]

⚠️ No Changeset found

Latest commit: 2929d68

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR