Code coverage for JavaScript tests

[mgeisler]

Now that #2462 is fixed, a natural next step would be to add support for code coverage on the JavaScript tests. Basically, I would like to know that we exercise most or all of book.js in our tests. The same goes for the other JavaScript files we maintain.

@Alx-Lai, @michael-kerscher or someone else, please let me know if you're interested in this.

[michael-kerscher]

I'm not sure if this is a major blocker but I wanted to document this at least.

We use the local runner

comprehensive-rust/tests/wdio.conf.ts

Line 8 in 380dc3c

runner: "local",

and we use the browser api to let the local runner test properties in the remote browser. Example:

comprehensive-rust/tests/src/basic.test.ts

Lines 6 to 7 in 380dc3c

	await browser.url("/");
	expect(await browser.execute(() => window.default_theme)).toBe("light");

I found documentation for coverage support for wdio but this documents how a browserrunner needs to be configured.

In order to enable code coverage reporting, enable it through the WebdriverIO browser runner configuration, e.g.:

This page also only documents Coverage Options for the browser runner

[michael-kerscher]

I'm not sure how to use the browser runner as this seems to requires a vite setup, which seems non-trivial for our use case. As far as I currently understand, vite needs to serve the entire html book + javascript and the tests will not run "outside" the served html but are injected into this site.

My current setup is looking at the website from the outside via the webdriver protocol with a browser that has this page open. The test is not running in the browser.

[michael-kerscher]

to document some new findings I have made: I have found https://webdriver.io/docs/devtools-service/#capture-code-coverage which can be used to get a coverage report via the devtool protocol but this might be phased out https://webdriver.io/blog/2024/08/15/webdriverio-v9-release/#removal-of-devtools-and-wdiodevtools-service-packages...

I nevertheless did a quick and dirty POC and it created a coverage report (formats are html, lcov, json or text). I opened a bug at https://github.com/webdriverio/webdriverio to clarify the documentation as it is not clear to me if a local runner will have this capability (without devtool service) in the future.

To try this for yourself you can add this to the services section in wdio.conf.ts

 [
      "devtools",
      {
        coverageReporter: {
          enable: true,
          type: "html", // lcov, json, text
          logDir: "coverage",
          exclude: [/resources/],
        },
      },
    ]

and also it seems to need an execution of await browser.getCoverageReport() to send the coverage report from the (remote) browser

[michael-kerscher]

update: webdriverio/webdriverio#12300 removed wdio/devtools-service which is replaced by https://webdriver.io/docs/lighthouse-service/ which is not featuring coverage support as far as I see.

The local browser runner is not planned to support coverage tests webdriverio/webdriverio#14115

[michael-kerscher]

not feasible with the current framework and setup.