[Feature] expose @playwright/test runner API

[bgotink]

We're currently using jest as a test runner for testing angular applications using playwright. In the angular ecosystem tools usually integrate with the @angular/cli by providing a builder, so our @ngx-playwright/jest package is actually what powers the ng e2e command in our repositories.

It would be great if we could write such an integration for the @playwright/test runner as well, but that'd require access to the runner APIs.
While it would be possible to execute the playwright CLI in a new process, this would make certain integrations harder or hacky. For instance passing configuration from the angular.json into the test runner would require turning those into CLI arguments and passing the base URL the angular app is hosted at into the runner would require a workaround using the child process's environment to pass this along.

[dgozman]

Is there an example of what this API would look like? I see that you'd like to pass "base URL", so this seems like somewhat deeper integration than just "run tests based on this configuration file". Maybe you could point to your existing integration with jest for us to take a look?

[bgotink]

Jest has quite a lot of extension points, our test integration provides a TestEnvironment, which then sets up the global environment the test has access to (e.g. we provide page, browserContext, browser etc as globals). This environment
itself can access the jest config, which contains the base URL.
The test environment can be found here: https://github.com/bgotink/ngx-playwright/blob/main/packages/jest/src/environment.ts

Jest actually has a configuration value called "testURL" they use in their jsdom test environment, we reused that one and pass it to jest here: https://github.com/bgotink/ngx-playwright/blob/main/packages/jest/src/builder/builder.ts#L54-L64

For @playwright/test things could probably be a lot easier, as the test runner already takes care of setting up the entire browser, context and page.
A function where we pass in the configuration (from the config file) and overrides (the CLI options) which then gets executed goes a long way to providing the necessary API. A separate function to read the config, so the caller of the test API doesn't have to tackle typescript conversion, would be useful to add.
This setup would allow us to add an extra fixture with the base URL in it to the tests, giving our other fixtures access to this variable.

If a watch-mode would be added to the test runner, being able to hook into that and rerun the tests when the angular app finishes rebuilding on a change would be nice to have.

[dgozman]

@bgotink Thank you, we'll take a look.

[bgotink]

The addition of the baseURL fixture made implementing this via child_process actually very easy to do. We've now switched from jest to playwright-test and it has been a breeze.

I still think an API would be better in the long run, as it would allow for more interesting integrations. For example, if a watch mode is implemented (#7035) an API could provide a hook to trigger a new run which we could use to re-run tests after every change to the application source. That would be impossible in the child_process approach, as we would have to restart playwright in its entirety and lose the context of the watcher (e.g. if the watcher supports things like jest's interactive UI the state the user selected would be lost)

[ryanbas21]

Hi,
Just wanted to say this would be really nice to have. I'm going to take the above approach and spawn a child process, but it would be great if playwright exposed something like jest does with runCLI for example. Anyway where we can pass in those values would help the community make playwright extensible.

I'm currently trying to write a plugin to use playwright with the playwright runner in an nx monorepo, and having this would make it clean and easier to implement i'd think. I scoured the codebase for awhile but couldn't find anything until reaching this issue

[bgotink]

@ryanbas21 FYI that's my exact use case as well, you might want to take a look at https://github.com/bgotink/ngx-playwright/tree/main/packages/test, even if only as inspiration.

[ryanbas21]

@dgozman any chance you guys explored this? The playwright runner is just so much faster than using Jest. I'd also love to be able to utilize the project command in my CI with nx so i can run my CI tests based on browser

[dgozman]

@ryanbas21 This is on our radar. It opens a quite large API surface, so we are being careful with still working out the balance between the flexibility and long-term commitment.

It seems like you'll be fine with run(args: string[]), or do you want more control? I don't see much value in just wrapping child_process.execSync() or similar 😄

I guess you'd like to pass something like FullConfig instead of creating a config file on disk and and/or constructing command line arguments yourself? The more details you provide, the more likely we'll be able to satisfy your usecase. No specific promises yet though.

[ryanbas21]

Hey @dgozman!

My use case is very similar to how this person wrote the jest playwright plugin for nx.

https://github.com/Bielik20/nx-plugins/blob/master/packages/nx-jest-playwright/src/executors/jest-playwright/executor.ts

Ideally, i would either use or write myself an nx plugin which executes my playwright tests in the given nx project.

I think FullConfig is what I want, i'd love to be able to pass in those configs and overrides in my configuration settings. In order to write that plugin, I really need something like jest exposes with runCLI.

Right now, all i'd really be doing is spawning a child process and executing npx playwright test -c myconfigfile and probably constructing some other CLI arguments from the nx configuration.

So - I want to use my config file on disk, but my configuration can provide some env overrides, like say CI settings may differ from local.

Exposing the runner so that I can pass a cli argument to it like --project chromium for CI would also be great.

I hope that brain soup helped a bit more.

Edit:
Another option would be just supplying that sort of nx plugin from your own repo if you don't want to expose too much of your internal API. Seems both myself and @bgotink ran into the same problem with nx. There is actually an open issue with nx repo to support Playwright but i'd assume even if they wanted to, they'd run into this problem as well.

I understand this is probably not as much of a solution than the above though.

[uavdrip]

Can anyone share an example as to how to run playwright test via process?

[jeremytenjo]

@uavdrip you can use shelljs

runPlaywright.js

const shell = require('shelljs')

function runPlayWright() {
 shell.exec('npx playwright test')
}

https://github.com/shelljs/shelljs

[uavdrip]

Thank you, I hope it would work on aws lambda.

[munichbughunter]

@uavdrip
You can run Playwright on AWS Lambda with this approach:

exports.lambdaHandler = async (event, context) => {

    const child_process = require('child_process');
    try {
        const result = child_process.spawnSync('npx', ['playwright', 'test', '__tests__/', '--reporter', 'list', '--config', 'playwright.config.js'], { encoding: 'utf8' });
        console.log('Execution Result: \n', result.stdout);

        response = {
            statusCode: 200,
            body: JSON.stringify({ message: result.stdout }),
        };
        return response;
    } catch(err) {
        console.log(err);
        return err;
    }
};

[dianaarce94]

Is it possible to run a child process with something like
child_process.exec('node mypath@playwright\test\cli.js test') to execute the playwright run?

[hav3n]

I'd like to +1 this as well, we're currently forced to do this via shelljs as well, funnily enough we use commander too so its commander all the way down + I don't love the idea of passing possibly untrusted strings to shelljs.exec.

Sample:

async function runPlaywright(options: PlaywrightRunOptions): Promise<void> {
  // Note that rootDir is relative to location of this file
  // It will need to be changed if this is moved elsewhere
  const rootDir = path.join(__dirname, '..', '..', '..');
  const binPath = path.join(rootDir, 'node_modules', '.bin');

  // Append node_modules/.bin path to current path, so we can run `playwright xyz` without prefixing
  process.env.PATH += path.delimiter + binPath;
  // Append node to current path, playwright has /usr/bin/env node shebang so we need to override
  process.env.PATH += path.delimiter + options.nodePath;

  const cmd = buildCommand(options, rootDir);

  console.log('Running command', cmd);

  exec(cmd);
}

Logically you could "just expose" the commander program directly and we could call it via await program.parseAsync('test', {project: 'xyz', config: 'config_path'}) but I can see why you wouldn't want to do that.

[RunOrVeith]

We would also be very interested in this feature, but from a different perspective:

We are dynamically generating test code, and if we could just do something like

const playwrightTestGenerator = (input: OurInput) => {
    return test.describe("generated", async () => {
        const page = ...;
        // Some other generated code that checks what we want
    })
};
//                      ------------- This is what we would need
//                      |
const output = playwrightRunner.run([
    playwrightTestGenerator("ourInput"), 
    playwrightTestGenerator("ourOtherInput")
]);

it would solve the problem of having to generate the code for the tests as actual typescript code that we can dump to a file, because we could just dynamically build what we need from within the testcase and our input.

Edit: How does the playwright vscode extension do this to run a specific test?

[brian-pickens]

@ryanbas21 This is on our radar. It opens a quite large API surface, so we are being careful with still working out the balance between the flexibility and long-term commitment.

It seems like you'll be fine with run(args: string[]), or do you want more control? I don't see much value in just wrapping child_process.execSync() or similar 😄

I guess you'd like to pass something like FullConfig instead of creating a config file on disk and and/or constructing command line arguments yourself? The more details you provide, the more likely we'll be able to satisfy your usecase. No specific promises yet though.

Being able to run tests simply with something as simple as a Run(FullConfig) method would be a great start.

[seebeen]

Hallo.

Would it be possible just to export some of the classes / functions. That would give me and my frens enough leeway to run Playwright programatically :)