diff --git a/tests/PLAYWRIGHT.md b/tests/PLAYWRIGHT.md new file mode 100644 index 000000000000..37c8e16d0bd7 --- /dev/null +++ b/tests/PLAYWRIGHT.md @@ -0,0 +1,198 @@ +# Headless tests with Playwright + +There are currently 3 general automated tests: + +1. `jest` tests against real English content (and some code) +1. `jest` tests against fixture content +1. `playwright` tests against fixture content (What this document is about!) + +## Quickstart + +Just like with regular `jest` tests, if you haven't already done so... + +```sh +npm run build +``` + +Now, to run all the tests: + +```sh +npm run playwright-test +``` + +That command will automatically start a server (on `localhost:4000`) for +the duration of test suite. It then finds all `tests/**/*.spec.ts` +files and run them by using `Google Chrome` as the underlying browser. + +If you have [set up a local Elasticsearch server](../search/elasticsearch-locally.md) (`localhost:9200`) the +headless tests will test doing site-searches if you've set up +an `ELASTICSEARCH_URL` environment variable. + +## Introduction + +The best documentation is and this documentation +here is merely an introduction to it. + +Refer to it when writing tests and trying to figure out how to use certain +[locators](https://playwright.dev/docs/locators) which is important +things, like `page.getByAltText()`, which you'll need for tying the browsing +to your assertions. + +### What to test + +Beyond some basic happy path tests, **only test what `jest` can't test**. +In particular this means client-side JavaScript interactions. For example, +`jest` can fetch the HTML over HTTP and assert against the `cheerio` parsed +HTML, but it can't test what happens when you click a client-side routing +link that triggers some sort of user agent interaction. + +`jest` is always faster. Playwright tests can test things like displaying +different things depending on cookies or `localStorage`. Playwright tests +can test the visual presence of something. For example, if something +like `
Text here
` is in the DOM only +Playwright can understand that it's not actually present in the page since +`jest` and Cheerio can't understand CSS. + +Think of your headless tests as "What would a human QA person do?" +The imaginary QA person can be you. If there's something you find yourself +doing to make sure your functionality doesn't regress as it's changing, +consider that to be motivation enough to write a headless test. + +## VSCode + +["Playwright Test for VSCode"](https://marketplace.visualstudio.com/items?itemName=ms-playwright.playwright) +is a great extension for people who use VSCode. Once installed, open the +file `playwright-rendering.spec.ts` and start the command +palette (`Cmd`+`Shift`+`p`) and type "Testing: Focus +on Playwright View" which will display the "TESTING" sidebar. It finds +all the file's tests as a tree and for each test there's a Play button. +You can either play a specific single test or you can make it run all +tests. + +Note, that failiure is often the result of the Playwright test waiting +very patiently for something to be present but it can't be found. I.e. +failures are often the same thing as Playwright reaching a waiting timeout. +This can make it feel like nothing's happening. + +Near the bottom of the "TESTING" sidebar is an extra menu specifically for +Playwright. One very useful option is the "[ ] Show browser" which means a +browser window will appear when tests run. + +## CLI + +The most basic command is: + +```sh +npm run playwright-test -- --help +``` + +This will guide you to all the options possible. For example, + +```sh +npm run playwright-test -- --headed +``` + +...will open a browser flickering through the tests. + +```sh +npm run playwright-test -- playwright-rendering.spec.ts +``` + +...will only run the tests in a file by that name. + +```sh +npm run playwright-test -- playwright-rendering.spec.ts:16 +``` + +...will run that specific `test('description here', async ({ page }))` on +line 16. + +```sh +npm run playwright-test -- -g "view home page" +``` + +...will only run tests whose description contains that text. + +## Debugging + +Writing tests can be difficult until all the locators feel like +second nature. You might be struggling with finding something in the +page which you're not sure is there or you don't know exactly +how to refer to it. + +The first thing to do is familiarize yourself with how to run the CLI +that only opens the one specific test you're debugging. Then, you +run the CLI with `--debug --headed`. For example: + +```sh +npm run playwright-test -- -g "view home page" --debug --headed +``` + +Now, it should open an additional debugger window next to a browser +window and you can press the play button there. When it gets stuck you can +use the browser window to do things like right-clicking and "Inspect..." +to understand what's in the DOM. + +Another thing that can help debugging is to open the browser just like +the script does. Run: + +```sh +npm run start-for-playwright +``` + +and open your regular browser window on . +When you're done, don't forgot to stop the server otherwise +the `npm run playwright-test` command won't work. + +## Codegen + +Codegen is when Playwright starts a browser and a debugger window. In the +debugger window it generates TypeScript code which you can copy-and-paste +into your editor/IDE when you're done. To use codegen you need to +first manually start the server. In the **first terminal**: + +```sh +npm run build && npm run start-for-playwright +``` + +In a **second terminal**: + +```sh +npx playwright codegen +``` + +Now type in `localhost:4000` in the browser window and click around. +Note how the TypeScript code gets written. It's definitely not perfect +but it can save you a lot of time writing selectors. + +Note that the codegen code will not have any assertions other than +sheer presence. It might also contain things like +`await page.goto('http://localhost:4000')` which you can later +correct to `await page.goto('/')`. + +When you have pasted over the TypeScript code from the debugger window, +you can click into that second terminal and press `Ctrl`+`C` to stop +the codegen debugger. + +## More browsers + +At the moment (March 2023) we don't test more browsers in Actions. +The primary use case at the moment is testing that client-side +interactions work at all. Actual cross-browser testing is not a priority +at the current time. + +## Tips on writing tests + +- What would a human be able to assert? If you find yourself testing things +that you expect in the DOM that a human wouldn't be able to test, the +test might not be a good test. For example, to make an assertion that +a certain div has `class="blabla"` if you click on a certain thing. Either +test something visual or perhaps don't bother testing it with Playwright. + +- *Combine* codegen tests and manual editing is a great combination. +Use the codegen output but familiarize yourself with the Playwright +documentation how to do things like locators and/or assertions. + +- When you use the codegen, it's clever in that it can attach to `data-testid` +nodes in your DOM. That's a good thing. If it's unable to do that, +consider going into the React code and add some more. diff --git a/tests/README.md b/tests/README.md index 8af3f5ffbbaf..b1abc59c5c33 100644 --- a/tests/README.md +++ b/tests/README.md @@ -120,3 +120,7 @@ jest tests/rendering/ -b ### Fixture based testing See [Fixture content](./fixtures/README.md). + +### Headless tests with Playwright + +See [Headless tests with Playwright](./PLAYWRIGHT.md) diff --git a/tests/meta/orphan-tests.js b/tests/meta/orphan-tests.js index ea7adb4be744..6be4dc4ee6ae 100644 --- a/tests/meta/orphan-tests.js +++ b/tests/meta/orphan-tests.js @@ -5,8 +5,8 @@ import path from 'path' describe('check for orphan tests', () => { test('all tests are in sub-directories', async () => { // A known list of exceptions that can live outside of directories - const EXCEPTIONS = ['README.md', 'package.json', 'utils.js', '.DS_Store'] - const pathToTests = path.join(process.cwd(), 'tests') + const EXCEPTIONS = ['README.md', 'package.json', 'utils.js', '.DS_Store', 'PLAYWRIGHT.md'] + const pathToTests = 'tests' // Get a list of files/directories in `/tests` const testDirectory = await fs.readdir(pathToTests)