[Feature] Track Code Coverage

[udit0802]

It would be great if playwright can provide OOTB support for calculating the code coverage of e2e tests.
This feature is provided by every test framework, as we are migrating to playwright from wdio, this would be a great help if this feature is available.

[aslushnikov]

There's a guide on how to setup test coverage: https://github.com/mxschmitt/playwright-test-coverage

Does it help?

[udit0802]

https://github.com/mxschmitt/playwright-test-coverage

I have tried that out, .nycoutput folder is generated but no reports are generated.
It did not help out much as this is react based, whereas my application is not.
I would highly appreciate, something OOTB can be provided by playwright team.
Looking forward for a positive response

[pavelfeldman]

How do you use this coverage data? (Coverage is typically used in the unit test context only, rarely used for e2e).

[udit0802]

How do you use this coverage data? (Coverage is typically used in the unit test context only, rarely used for e2e).

Very similar use case as mentioned here.
#7030 (comment)

[udit0802]

Hi any plans/ETA for this feature request?

[udit0802]

@aslushnikov
Gentle Reminder!!
Any update on this feature request?

[udit0802]

Gentle Reminder!!
Any update for the same.
Any help would be highly appreciated.

[p01]

Code coverage of the final bundles would be quite useful to know how much ground the E2E tests cover and to assess whether a new test add sufficient value compared to the cost of running on CI in a big a repo.

[Uninen]

The guide uses babel-plugin-istanbul which is not helpful for modern projects that don't use babel (like Vite / esbuild). It would be nice to have a way to use for example c8 instead.

[jfgreffier]

Playwright actually offers API to get v8's coverage. There is an example using v8-to-istanbul, but it is rather incomplete since it doesn't allow generating coverage report.
https://playwright.dev/docs/api/class-coverage

You can indeed get coverage from v8 and read it from c8. Here is what I did :

const { test, expect } = require("@playwright/test");
const fs = require("fs");
const url = require("url");
const path = require("path");

test("test", async ({ page }) => {
  // start coverage
  await page.coverage.startJSCoverage();

  // TODO Your test
  ...

  // Get and save V8 coverage
  const coverage = await page.coverage.stopJSCoverage();
  const rootPath = path.normalize(`${__dirname}/..`);
  const coverageWithPath = coverage.map((entry) => {
    const fileName = new url.URL(entry.url).pathname;
    return { ...entry, url: `file:///${rootPath}${fileName}` };
  });

  fs.writeFileSync(
    "coverage/tmp/coverage.json",
    JSON.stringify({ result: coverageWithPath }, null, 2)
  );
});

I had to fix the url, by getting the full path of the files. You might need to adjust rootPath.

After that, just run npx c8 report

[merishabhgupta]

Hi
I don't have much experience with testing. So my query is how to get code coverage of e2e tests written with playwright.
Scenario:
We have a different repo for automation test cases which contains only the automation test cases and they point to some instance where the production application is deployed. So we are running automation test cases on local over a deployed application on some server.
We have to get the code coverage from these e2e tests when we will be running these tests on the same server after deploying this test cases repo on the same server as our application.
How should the config look in this scenario.
Any kind of help is appreciated.
@aslushnikov
@jfgreffier
@udit0802

[testgitdl]

@jfgreffier I tried using your code but I do not get any coverage information. I am starting my app via "ng serve -c dev" and afterwards want to get coverage information.
I have an Angular 14 project with ESM and the test framework is a cucumberjs + Playwright + TypeScript.
I start and stop jscoverage in the beforeall/afterall hooks but do not receive any info although the coverage.json is created.
Do you know what I am doing wrong?

hOPE YOU CAN HELP
``
BeforeAll(async function () {
browser = await chromium.launch(configChrome);
context = await browser.newContext();
page = await context.newPage();
await page.coverage.startJSCoverage();
});

AfterAll(async function () {
const __filename = fileURLToPath(import.meta.url);
const __dirname = path.dirname(__filename);
const coverage = await page.coverage.stopJSCoverage();
const rootPath = path.normalize(${__dirname}/..);
const coverageWithPath = coverage.map((entry) => {
const fileName = new url1.URL(entry.url).pathname;
return { ...entry, url: file:///${rootPath}${fileName} };
});
fs.writeFileSync(
"coverage/tmp/coverage.json",
JSON.stringify({ result: coverageWithPath }, null, 2)
);
``

Output of npx c8 report
----------|---------|----------|---------|---------|-------------------
File | % Stmts | % Branch | % Funcs | % Lines | Uncovered Line #s
----------|---------|----------|---------|---------|-------------------
All files | 0 | 0 | 0 | 0 |
----------|---------|----------|---------|---------|-------------------

[kachkaev]

For those who are looking for a temp drop-in solution, I can recommend anishkny/playwright-test-coverage, which is available on NPM. It uses Istanbul instead of C8, which allows to collect coverage from all browsers. Before running tests, the app needs to be built with coverage instrumented (via babel-plugin-istanbul).

We use playwright-test-coverage package in blockprotocol/blockprotocol inside integration tests (five browsers). The stats are uploaded to Codecov. Search for TEST_COVERAGE and playwright-test-coverage for hints.

Hope to see built-in coverage reporting at some point!

[testgitdl]

@kachkaev thank you very much for checking. I already took this as an example and the problem is that the nyc-output is empty - probably it has to do with the nyc ng serve. Please let me know if you have more ideas - would help big time
Below my npm scripts:
"serve": "nyc ng serve", "test:only": "playwright test", "test": "start-server-and-test serve http://localhost:4200 test:only"

And the output of npm run test

new-project@0.0.0 test
start-server-and-test serve http://localhost:4200 test:only

1: starting server using command "npm run serve"
and when url "[ 'http://localhost:4200' ]" is responding with HTTP status code 200
running tests using command "npm run test:only"

new-project@0.0.0 serve
nyc ng serve

✔ Browser application bundle generation complete.

Initial Chunk Files | Names | Raw Size
vendor.js | vendor | 1.77 MB |
polyfills.js | polyfills | 318.06 kB |
styles.css, styles.js | styles | 210.09 kB |
main.js | main | 48.04 kB |
runtime.js | runtime | 6.52 kB |

                  | Initial Total |   2.34 MB

Build at: 2022-11-18T14:33:43.337Z - Hash: 0380aa086066a4e4 - Time: 5119ms

** Angular Live Development Server is listening on localhost:4200, open your browser on http://localhost:4200/ **

√ Compiled successfully.

new-project@0.0.0 test:only
playwright test

Running 3 tests using 3 workers

3 passed (7s)

To open last HTML report run:

npx playwright show-report

[testgitdl]

@kachkaev I have created a sample project here: https://github.com/testgitdl/playwright-coverage

[jfgreffier]

@testgitdl for c8 coverage, yoiu should open the coverage.json and check it. You might have coverage info, or not. Maybe you have coverage info but the wrong path... You should come over to the Slack, a GitHub issue isn't very convenient

[jennydaman]

@jfgreffier provided a great example in #9208 (comment)

I've improved on his code snippet by putting it into a fixture.

// fixtures.ts
import { test as base } from "@playwright/test";
import * as fsPromises from "node:fs/promises";
import * as path from "node:path";

type MyFixtures = {
  _withCoverage: undefined;
};

export const test = base.extend<MyFixtures>({
  _withCoverage: async ({ page }, use, testInfo) => {
    await page.coverage.startJSCoverage();
    await use(undefined);

    const coverage = await page.coverage.stopJSCoverage();
    const srcPath = "@fs" + path.normalize(`${__dirname}/../../src`);
    const srcCoverage = coverage
      .filter((entry) => entry.url.includes(srcPath))
      .map((entry) => {
        return { ...entry, url: entry.url.replace(/^.+@fs/, "file://") };
      });
    await fsPromises.mkdir("coverage/tmp", { recursive: true });
    const testTitle = testInfo.title.replaceAll("/", "_");
    await fsPromises.writeFile(
      `coverage/tmp/${testTitle}.json`,
      JSON.stringify({ result: srcCoverage }, null, 2),
    );
  },
});

export { expect } from "@playwright/test";

Then in your test files:

import { test, expect } from "./fixtures.ts";

test("test1", async ({ page, _withCoverage }) => {
  await page.goto("http://localhost:5173/");
});

test("test2", async ({ page, _withCoverage }) => {
  await page.goto("http://localhost:5173/");
});

After running playwright test, the fixture creates coverage/tmp/test1.json and coverage/tmp/test2.json

[jfgreffier]

@jennydaman It is actually improving on my example ! Do you think you can make it into a Github repository for people to try it out and contribute ?

[jennydaman]

I couldn't figure it out how to do so. I wanted to try extending defineConfig and test in a separate package following examples from https://github.com/microsoft/playwright/tree/main/packages/playwright-ct-core and https://github.com/anishkny/playwright-test-coverage however I kept getting errors saying

Error: Requiring @playwright/test second time,

EDIT: I found the workaround here #15819, will create the package!

[jennydaman]

I have developed a minimal package which configures Playwright to collect test coverage and write the data to a directory.

https://www.npmjs.com/package/playwright-test-coverage-native