[Feature] "Headless" CLI Watch Mode

Playwright recently introduced a watch mode via a new --ui flag. While awesome, it doesn't quite meet some other use cases for a "Watch Mode". Namely continuous background validation of tests where a UI might be intrusive, but you still want red/green when test srcs are updated. At that point you might jump in the UI to debug, but by default you're mostly just interested in continuous background validation.

Playwright is also increasingly being used for unit tests or integration tests, where a full browser step through might not be relevant.

Try setting environment variable PWTEST_WATCH=1 and running your tests. It'll engage a different version of the watch mode that should look more like the traditional jest one. Tell us what you think about it and whether it worked for you. We'll collect upvotes here to see if we want to make it official --watch.

While we all waiting on this there's also a way to run a watch mode if you're on CI, using the git staged files. You can pull the files changed in git and feed to Playwright CLI to run only the specs you want. Here's an example with Github Actions:

steps:
  - name: Checkout
    uses: actions/checkout@v3

  - name: Setup NodeJS
    uses: actions/setup-node@v3
    with:
      node-version: 16

  - name: Install dependencies
    run: npm install

  - name: Install latest playwright browsers
    run: npx playwright install --with-deps

  - name: Get changed specs in the src folder
    id: changed-files-folder
    uses: tj-actions/changed-files@v35
    with:
      files: folder/**

  - name: Test 'e2e' if folder files changed
    if: steps.changed-files-folder.outputs.any_changed == 'true'
    run: npx playwright test ${{ steps.changed-files-folder.outputs.all_changed_files }}

at the end, the above would issue a command like

npx playwright test example.spec.ts second.spec.ts

I really love PWTEST_WATCH=1 with component testing, it makes the experience more enjoyable than UI Mode, especially if you attempt to do TDD. Also the possibility to see the actual browser with the real component (instead of Trace) is a plus compared to the UI Mode.

feat: use experimental watch mode from microsoft/playwright#21960 (co…

This only watches test files. Especially with component testing which are basically unit tests and follow short "Red light - Green light - Refactor" loops, it would be very useful if the CLI watched source files also the way Jest's VS Code extension does.

This only watches test files. Especially with component testing which are basically unit tests and follow short "Red light - Green light - Refactor" loops, it would be very useful if the CLI watched source files also the way Jest's VS Code extension does.

That's what it does.

Pavel Feldman (@pavelfeldman)

UPDATE: Now using onchange with -k option instead of chokidar. If there's a file change while tests are already running in response to a previous change, this would kill running tests and restart. chokidar on the other hand would launch tests another time in parallel.

This is the new test-playwright:watch script:

"test-playwright:testcommand": "(playwright test quiz.spec.ts || true) && echo Test run report available at http://localhost:6007",

"test-playwright:watch": "concurrently --names \"REPORT,TEST\" --hide REPORT --prefix-colors \"blue,magenta\" \"http-server playwright-report -s --port 6007\" \"onchange -i -k tests/**/*.spec.ts pages/* components/**/* -- npm run test-playwright:testcommand\"",

Hi Pavel Feldman (@pavelfeldman),

I am pretty certain that watch mode of playwright test only monitors test files. I couldn't trigger it by making changes to any of my code files (React components) and saving the files. I also couldn't find any command line option to make it watch non-test code.

Another thing it doesn't do is create or serve HTML report that can be really useful when there are failures.

I am currently using a workaround using chokidar, playwright test, http-server etc. (given below). It would be great if the watch mode built into playwright test could incorporate these two features.

My current watch mode setup is as follows: I have a script named test-playwright:watch in package.json to do both (i.e. watch for changes to both test and code-under-test files, and to serve HTML report). It uses another two scripts and all three are shown below:

"test-playwright-watch-testcommand": "(playwright test quiz.spec.ts || true) && echo Test run report available at http://localhost:6007",

"test-playwright-watch-watchcommand": "chokidar tests/**/*.spec.ts pages/* components/**/* -c \"npm run test-playwright-watch-testcommand\"",

"test-playwright:watch": "concurrently -k -s first -n \"REPORT,TEST\" -c \"magenta,blue\" \"http-server playwright-report -s --port 6007\"  \"run-s --silent -c test-playwright-watch-testcommand test-playwright-watch-watchcommand\"",

Since every run of playwright test opens up a new HTTP server on the playwright-report directory (although by default the report is only served on failure), a lot of these would end up running at the same time (each on a different, probabaly randomly assinged, port but on the same underlying folder playwright-report) after tests have run enough time due to file changes detected by chokidar. Therefore, I configure the HTML reporter in playwright.config.ts to generate the HTML report but not to serve it:

reporter: [['html', { open: 'never' }]],

and instead launch http-server on the report directory only once at the time the script is launched. I do this by running the actual watch/test command in parallel with http-server using concurrently (see definition of test-playwright:watch above).

Then at the end of every run of playwright-test that had started on detection of a file change, the URL of the report is printed so you can just Ctrl+Click it to look at the report in a browser window:

To get the above scripts to work, I think the following NPM packages would need to be installed (in addition to playwright of course):

npm install -D concurrently http-server chokidar npm-run-all

Finally, for anyone using VS Code who wants test-playwright:watch script to start running automatically whenever the project folder is opened, they can define a task in .vscode/tasks.json like this (full tasks.json shown so you can just copy and paste it in a blank .vscode/tasks.json if you don't have one already). Note for this to work you would also need to configure webServer in playwright.config.ts to launch your app whenever tests start running. This is shown after tasks.json below.

{
  // See https://go.microsoft.com/fwlink/?LinkId=733558
  // for the documentation about the tasks.json format
  "version": "2.0.0",
  "tasks": [
    {
      "label": "Playwright watch",
      "type": "shell", //"process" does not work as value
      "command": "npm run test-playwright:watch",
      "args": [],
      "isBackground": true, //this makes it keep running in the background without interrupting what you're doing
      "runOptions": {
        "runOn": "folderOpen" //start this task when folder is opened in VS Code
      },
      "presentation": {
        "panel": "dedicated"
      },
      "problemMatcher": {
        "fileLocation": ["relative", "${workspaceFolder}/tests"],
        "severity": "error", //probabaly redundant as it is the default
        "pattern": {
          "regexp": "^\\[TEST\\]\\s*\\[(.*)\\]\\s\\S\\s(.*):(\\d+):(\\d+).*$",
          "file": 2,
          "line": 3,
          "column": 4,
          "message": 1

        },
        "background": {
          "activeOnStart": false,
          //I have verified that if either of the following regexes is wrong,
          //you don't get the tick on the terminal to indicate once
          //a test run has completed (even though our watch task carries on running)
          "beginsPattern": "^\\[TEST\\]\\s*Running \\d* tests using \\d* workers\\s*$",
          //"beginsPattern": "^\\s*Running \\d* tests using \\d* workers\\s*$",
          //USE THE FOLLOWING endsPattern IF REVERTING TO CHOKIDAR TO WATCH
          "endsPattern": "^\\[TEST\\]\\s*To open last HTML report run:\\s*$"
          //"endsPattern": "^Waiting for file changes\\. Press enter to run tests, q to quit or h for more options\\.\\s*"
        }
      }
    }
  ]
}

In playwright.config.ts, configure webServer sort of like this:

webServer: {
    command: 'npm run dev',
    url: 'http://127.0.0.1:3000',
    reuseExistingServer: !process.env.CI,
    ignoreHTTPSErrors: true,
  },

With the three patterns in problemMatcher in the task (regex in pattern block, beginsPatter and endsPattern), the VS Code background task provides a nice indication of its status in the header of the terminal window in which it is running:

If there are failures, you can see indications in a few different places:

terminal window label (shown above)
The terminal itself (if in view), where you can see the failing tests listed and Ctrl+Click URL of the test run report to open it in a browser (also shown above)
In Problems pane, from where you can also navigate directly to the the test that failed:

Everybody: are you still using this mode? How do you feel about us removing this experiment? Do you feel like it's adding enough value atop of chokidar?

Let us know what you think!

Dear Andrey, I don’t think it’s adding any value as the UX is not there. A watch mode should be supplemented with visual indicators of failure and launch on opening the folder in the editor of the sort you get when you run the watch command from a VS Code task in .vscode/tasks.json. Also, the mode doesn’t generate or serve Playwright’s marvellous HTML report. I had to stitch that together with concurrently, http-server and plain “npm playwright test –reporter=html”. See my comment below if you haven't already; this I think is a fairly complete watch mode with a good UX: #21960 (comment) One place where you could add lots of value I think is if you could provide a command in Command Pallette to generate both the watch command in package.json and an accompanying task in .vscode/tasks.json. Writing the whole thing for the first time can be real pain because: * some patterns that regex101 excepts are not matched by VS Code. Playwright watch mode generates a line of text with a chevron in it that VS Code can't match. Figuring this out can be a pain. * In all my watch modes (Storybook, accessibility tests and Playwright tests) I have to echo out a beginning line of text and an ending line of text so that the problemMatcher in VS Code task can know when a tests run started and when it ended (crucial as the watch task itself runs perpetually). You could instead use echo these out to match you VS Code task's regex patterns. * There are lots of nuances of VS Code tasks that take a lot of trial and error to figure out the first time you write a watch mode. Bests, Naveed Sent from Mail <https://go.microsoft.com/fwlink/?LinkId=550986> for Windows *From: *Andrey Lushnikov ***@***.***> *Sent: *31 July 2023 19:05 *To: *microsoft/playwright ***@***.***> *Cc: *naveedausaf ***@***.***>; Comment ***@***.***> *Subject: *Re: [microsoft/playwright] [Feature] "Headless" CLI Watch Mode (Issue #21960) *Everybody*: are you still using this mode? How do you feel about us removing this experiment? Do you feel like it's adding enough value atop of chokidar? Let us know what you think! — Reply to this email directly, view it on GitHub <#21960 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AE3REKWEFACZPJJUAQISNWLXS7XWVANCNFSM6AAAAAAWGZ4ERE> . You are receiving this because you commented.Message ID: ***@***.***>

On second thoughts, it would be great if Playwright extension could just provide a command that could launch a similar watch mode in a separate terminal window with similar UX (indications of failure, successful completion and running tests in terminal header). The folders to watch in addition to `tests` could be configured in `playwright.config.ts`. It really is very useful to have it. Basically, iPlaywright is so fast and I have enough cores that as I am done with git add and git commit, I see a tick to say that all my tests have passed. I can then do git push: ![image](https://github.com/microsoft/playwright/assets/20386346/902a43bd-5396-4b93-8e9a-945626a71803) Thanks, Naveed

…

On Tue, Aug 1, 2023 at 1:33 AM Naveed Ausaf ***@***.***> wrote: Dear Andrey, I don’t think it’s adding any value as the UX is not there. A watch mode should be supplemented with visual indicators of failure and launch on opening the folder in the editor of the sort you get when you run the watch command from a VS Code task in .vscode/tasks.json. Also, the mode doesn’t generate or serve Playwright’s marvellous HTML report. I had to stitch that together with concurrently, http-server and plain “npm playwright test –reporter=html”. See my comment below if you haven't already; this I think is a fairly complete watch mode with a good UX: #21960 (comment) One place where you could add lots of value I think is if you could provide a command in Command Pallette to generate both the watch command in package.json and an accompanying task in .vscode/tasks.json. Writing the whole thing for the first time can be real pain because: * some patterns that regex101 excepts are not matched by VS Code. Playwright watch mode generates a line of text with a chevron in it that VS Code can't match. Figuring this out can be a pain. * In all my watch modes (Storybook, accessibility tests and Playwright tests) I have to echo out a beginning line of text and an ending line of text so that the problemMatcher in VS Code task can know when a tests run started and when it ended (crucial as the watch task itself runs perpetually). You could instead use echo these out to match you VS Code task's regex patterns. * There are lots of nuances of VS Code tasks that take a lot of trial and error to figure out the first time you write a watch mode. Bests, Naveed Sent from Mail <https://go.microsoft.com/fwlink/?LinkId=550986> for Windows *From: *Andrey Lushnikov ***@***.***> *Sent: *31 July 2023 19:05 *To: *microsoft/playwright ***@***.***> *Cc: *naveedausaf ***@***.***>; Comment ***@***.***> *Subject: *Re: [microsoft/playwright] [Feature] "Headless" CLI Watch Mode (Issue #21960) *Everybody*: are you still using this mode? How do you feel about us removing this experiment? Do you feel like it's adding enough value atop of chokidar? Let us know what you think! — Reply to this email directly, view it on GitHub <#21960 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AE3REKWEFACZPJJUAQISNWLXS7XWVANCNFSM6AAAAAAWGZ4ERE> . You are receiving this because you commented.Message ID: ***@***.***>

Andrey Lushnikov (@aslushnikov)

Andrey Lushnikov (@aslushnikov) I like this mode.

Even though I mostly use UI Mode, I still regularly use PWTEST_WATCH=1 on its own, without chokidar or anything, and it works like a charm for my purposes!

Some of our team members even prefer PWTEST_WATCH=1 over UI Mode and I suspect they would be sad to see this experiment removed.

Andrey Lushnikov (@aslushnikov)

Andrey Lushnikov (@aslushnikov) I use this mode for component testing, with and without showing the browser. Also, without browser, the experience is similar to Jest's --watch which feels confortable

Would love a native, traditional watch mode for Playwright. I typically write code in the terminal and use tmux for subwindows. With other test runners, I can have them run in a subwindow and whenever I modify a test or part of the code, it will rerun the affected tests and show me a red/green result quickly. I can't do that with Playwright. The --ui flag means switching to another window, hiding my text editor. I'd much rather have it all in my terminal without needing to switch. Basically something like PWTEST_WATCH=1 npx playwright test but that also reruns the relevant tests when I modify the associated code; not only the test files :)

Would it be possible to run only the tests related to the open Git changes when initiating tests using PWTEST_WATCH=1, rather than executing all of them right away? Jest/Vitest uses a similar approach.

Is it also an option to add test cancelation by, for example, typing p so i can immediately search for a single test and run it without having to wait till they all complete?

I really like PWTEST_WATCH=1
It's a very productive mode but I am facing an issue, and I'm not sure if it's a bug or intended purpose :

Basically when I use "Show & reuse browser: on"

Only my setup project runs in headful browser, afterwards all tests run headlessly, any one knows what could be the issue ?

Ideally, the CLI watch mode behaves exactly like vitest watch. Watch all file dependencies, only rerun the tests that change, provide some controls for initiating a rerun of (all|changed|pattern-matching) tests. That's what everyone here is missing.

Try setting environment variable PWTEST_WATCH=1 and running your tests. It'll engage a different version of the watch mode that should look more like the traditional jest one. Tell us what you think about it and whether it worked for you. We'll collect upvotes here to see if we want to make it official --watch.

PWTEST_WATCH=1 works perfectly.

Regardless of what the final solution will be it should be documented right away.

We'll collect upvotes here to see if we want to make it official --watch.

I give +1 for --watch. I've had to install cross-env just to be able to use PWTEST_WATCH=1, since it's the only script I use with an environment variable.

By the way, I think your comment had a lot of upvotes, so, what do you think?

This only watches test files. Especially with component testing which are basically unit tests and follow short "Red light - Green light - Refactor" loops, it would be very useful if the CLI watched source files also the way Jest's VS Code extension does.

That's what it does.

So what do you think could be the reason why when I use PWTEST_WATCH=1 without using fixtures like page or context the watch mode works much slower than Vitest and Bun:test?

Another +1 for watch. It's really important to have in a TDD workflow and just a good DX. Thanks for the PWTEST_WATCH=1 workaround - it's good but I can't wait for the day when Playwright will only rerun the tests I change.

Playwright CLI has an --only-changed flag now https://playwright.dev/docs/release-notes#--only-changed-cli-option

Im also a fan of PWTEST_WATCH=1 however, it being an environment variable as opposed to a CLI flag makes it difficult to work with my build scripts. A flag would be great

I am also using headless watch mode and like it but I am not able to add test coverage report for each test iteration. In CI I successfully using a global teardown for this. In watch mode I tried project level teardown. It is working as expected in CI headless mode (see 1st screenshot) but never called in watch mode (see 2nd screenshot) on each test iteration but run ones on quit. In jest similar setup working for each test iteration and it is very convenient.

  projects: [
    {
      name: 'code coverage report',
      testMatch: /project\.teardown\.ts/,
    },
    {
      name: 'chromium',
      use: { ...devices['Desktop Chrome'] },
      teardown: 'code coverage report',
    },

A --watch flag on the command line would be really great. It's one of the features I'm missing the most.

That aside, thank you so much for your work. Playwright is such a great tool!

Navigation Menu

Repository navigation

[Feature] "Headless" CLI Watch Mode #21960

5 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[Feature] "Headless" CLI Watch Mode #21960

Description

Activity

pavelfeldman commented on Mar 24, 2023

lucasassisrosa commented on May 6, 2023

jfgreffier commented on May 31, 2023

naveedausaf commented on Jul 11, 2023

pavelfeldman commented on Jul 13, 2023

nausaf commented on Jul 28, 2023

aslushnikov commented on Jul 31, 2023

naveedausaf commented on Aug 1, 2023

naveedausaf commented on Aug 1, 2023

bingocaller commented on Aug 1, 2023

jfgreffier commented on Aug 2, 2023

5 remaining items

coaxial commented on Aug 28, 2023

sand4rt commented on Aug 28, 2023

kenanAlhindi97 commented on Sep 21, 2023

mikestopcontinues commented on Dec 21, 2023

ildella commented on Feb 20, 2024

GermanJablo commented on Apr 30, 2024

kjmj commented on Sep 4, 2024

patrickberkeley commented on Sep 4, 2024

vinnythestick commented on Mar 14, 2025

VadimPassynkov commented on Jun 23, 2025

pbroschwitz commented on Oct 21, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions