Using lighthouse programmatically in typescript fails

[leadoflife]

FAQ

• Yes, my issue is not about variability or throttling.
• Yes, my issue is not about a specific accessibility audit (file with axe-core instead).

URL

https://demo.playwright.dev/todomvc

What happened?

Following the general instructions to use lighthouse programmatically...

... in a TypeScript project with Playwright tests...

... with both a top-level import of lighthouse as well as trying a dynamic import ...

... the test begins, but fails with an ES Module import error.

Error: require() of ES Module MY_PROJECT_PATH/node_modules/.pnpm/lighthouse@10.2.0_typescript@5.0.2/node_modules/lighthouse/core/index.js from MY_PROJECT_PATH/tests/performance.spec.ts not supported.
Instead change the require of index.js in MY_PROJECT_PATH/tests/performance.spec.ts to a dynamic import() which is available in all CommonJS modules.

What did you expect?

Either version of the import syntax to "just work".

What have you tried?

• Top level import with import lighthouse from 'lighthouse';
• Dynamic import with const lighthouse = await import('lighthouse');
• The completely separate npm package "playwright-lighthouse" (that seemed to have conflicts with both npm dependencies "playwright" and "@playwright/test" and executing playwright test seemed confused)

MY_PROJECT_PATH/tests/performance.spec.ts

import { test } from '@playwright/test';
import * as fs from 'fs';

test('performance report', async ({ page, browserName }) => {
  await page.goto('/');

  if (browserName === 'chromium') {
    const options = {
      logLevel: 'info',
      output: 'html',
      onlyCategories: ['performance'],
      port: 8041,
    };

    const lighthouse = await import('lighthouse');

    const runnerResult = await lighthouse.default(page.url(), options as any);

    // `.report` is the HTML report as a string
    const reportHtml = runnerResult!.report;
    fs.writeFileSync('./lighthouse-report.html', reportHtml);
  }
});

How were you running Lighthouse?

node, Other

Lighthouse Version

10.2.0

Chrome Version

"@playwright/test": "^1.34.3",

Node Version

v18.16.0

OS

Windows 11 Pro, 22H2

Relevant log output

Error: require() of ES Module MY_PROJECT_PATH/node_modules/.pnpm/lighthouse@10.2.0_typescript@5.0.2/node_modules/lighthouse/core/index.js from MY_PROJECT_PATH/tests/performance.spec.ts not supported.
    Instead change the require of index.js in MY_PROJECT_PATH/tests/performance.spec.ts to a dynamic import() which is available in all CommonJS modules.

[leadoflife]

I'll check if this error occurs using a version prior to the refactor with ES Modules from #12689

[leadoflife]

Did the port change in earlier versions from late 2022 like 9.6.8 or 9.6.7?

I got past the ES Modules error in the most recent version with both of those prior versions, but get the following error from both:

Error: connect ECONNREFUSED 127.0.0.1:8041

[connorjclark]

Did you try require('lighthouse/core/index.cjs')?

The error message you give suggests that your test file is being transpiled to commonjs before running, I think. You did a dynamic import and that should have worked, yet the error message says you didn't. What's your test runner?

[leadoflife]

Thanks @connorjclark , that got me past the module error to a new one below. I had seen another thread that suggested to use the port "8041" to connect to the Playwright port on Chromium. *I know this isn't the Playwright project or the right place for support, but am curious to know if the lighthouse experts already know what to do here...

TypeError: Failed to fetch browser webSocket URL from http://127.0.0.1:8041/json/version: fetch failed

  14 |     const lighthouse = require('lighthouse/core/index.cjs');
  15 |
> 16 |     const runnerResult = await lighthouse(page.url(), lighthouseOptions as any);
     |                          ^
  17 |   }
  18 |
  19 |   expect(true).toBe(true);

    at getWSEndpoint (MY_PROJECT_PATH/node_modules/.pnpm/puppeteer-core@20.5.0_typescript@5.0.2/node_modules/puppeteer-core/src/common/BrowserConnector.ts:160:20)
    at _connectToCDPBrowser (MY_PROJECT_PATH/node_modules/.pnpm/puppeteer-core@20.5.0_typescript@5.0.2/node_modules/puppeteer-core/src/common/BrowserConnector.ts:119:27)
    at MY_PROJECT_PATH/node_modules/.pnpm/lighthouse@10.2.0_typescript@5.0.2/node_modules/lighthouse/core/gather/navigation-runner.js:356:21
    at Function.gather (MY_PROJECT_PATH/node_modules/.pnpm/lighthouse@10.2.0_typescript@5.0.2/node_modules/lighthouse/core/runner.js:218:21)
    at navigationGather (MY_PROJECT_PATH/node_modules/.pnpm/lighthouse@10.2.0_typescript@5.0.2/node_modules/lighthouse/core/gather/navigation-runner.js:343:21)
    at navigation (MY_PROJECT_PATH/node_modules/.pnpm/lighthouse@10.2.0_typescript@5.0.2/node_modules/lighthouse/core/index.js:95:24)
    at MY_PROJECT_PATH/src/apps/leadof-us/tests/performance.spec.ts:16:26

[leadoflife]

There was an example provided in this comment:
microsoft/playwright#5956 (comment)

However, I'm using the "@playwright/test" dependency which is an abstraction over "playwright", which the example uses.

As a result, I configured playwright with the chromium flag to specify the debug port that lighthouse can use reliably. This configuration uses the same solution outlined in a Playwright issue comment about how to set chromium flags.

  /* Configure projects for major browsers */
  projects: [
    {
      name: 'chromium',
      use: {
        ...devices['Desktop Chrome'],
        launchOptions: {
          args: ['--remote-debugging-port=8041'],
        },
      },
    },
    ...

But, continue to get the error:

TypeError: Failed to fetch browser webSocket URL from http://127.0.0.1:8041/json/version: fetch failed

[leadoflife]

And, also tried this suggestion:
microsoft/playwright#2667 (comment)

Setting the environment variable prior to running the tests...
PLAYWRIGHT_CHROMIUM_DEBUG_PORT=8041

[leadoflife]

@connorjclark et al,

I added a debug call to the chromium debug endpoint by adding the below to the test.

    await page.goto('http://127.0.0.1:8041/json/version');

    console.log('content', await page.content());

... and getting the output...

content <html><head><meta name="color-scheme" content="light dark"></head><body><pre style="word-wrap: break-word; white-space: pre-wrap;">{
   "Browser": "HeadlessChrome/114.0.5735.35",
   "Protocol-Version": "1.3",
   "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/114.0.5735.35 Safari/537.36",
   "V8-Version": "11.4.183.8",
   "WebKit-Version": "537.36 (@2295354895fa3652ae47b651481831484f16d1ff)",
   "webSocketDebuggerUrl": "ws://127.0.0.1:8041/devtools/browser/b3cf8617-8966-44bc-beb1-1147ad88f845"
}
</pre></body></html>

💥 But, the lighthouse connection still fails with:

TypeError: Failed to fetch browser webSocket URL from http://127.0.0.1:8041/json/version: fetch failed

[leadoflife]

Update with extra info: We can run the CLI against an already running URL. But, we are hoping to integrate this into an automated test. Without the above Playwright integration working, we'll look into automating the ng serve and the lighthouse http://localhost:4200 CLI commands sequentially followed by logic to always clean up the ng serve process. Ideally, we'd love to avoid the CLI automation and process management with a clean Playwright test integration that already hosts the site and allows us to navigate and interact with it too.

[leadoflife]

Great news @connorjclark !

We got this working:
https://github.com/leadof/leadof/blob/eef64385862390923bcbc15c1e0ac70235d6bba9/src/apps/leadof-us/tests/performance.spec.ts