Lighthouse performance downgrade since 7.6.0

[Sybsw]

Current behavior

Cypress 7.6.0 and 7.7.0 have severely reduced performance when using cy.lighthouse(),
demonstrated here between 7.5.0 and 7.6.0:

✘ categories.performance failure for minScore assertion
all values 7.5.0: 0.97, 0.98, 0.98, 0.98, 0.97
all values 7.6.0: 0.82, 0.83, 0.72, 0.83, 0.83

⚠️ largest-contentful-paint warning for minScore assertion
all values 7.5.0: 0.88, 0.94, 0.94, 0.94, 0.88
all values 7.6.0: 0.49, 0.51, 0.07, 0.5, 0.49

✅ cumulative-layout-shift passing for minScore assertion
all values 7.5.0: 1, 1, 1, 1, 1
all values 7.6.0: 1, 1, 1, 1, 1

⚠️ first-contentful-paint passing for minScore assertion
all values 7.5.0: 0.98, 0.98, 0.98, 0.98, 0.98
all values 7.6.0: 0.63, 0.65, 0.66, 0.64, 0.65

⚠️ interactive passing for minScore assertion
all values 7.5.0: 1, 1, 1, 1, 1
all values 7.6.0: 0.98, 0.98, 0.98, 0.98, 0.98

⚠️ speed-index passing for minScore assertion
all values 7.5.0: 1, 1, 1, 1, 1
all values 7.6.0: 0.91, 0.92, 0.9, 0.92, 0.92

✅ total-blocking-time passing for minScore assertion
all values 7.5.0: 1, 1, 1, 1, 1
all values 7.6.0: 1, 1, 1, 1, 1

Desired behavior

lighthouse tests should perform at the same speed they did before (7.5.0)

Test code to reproduce

context('perf', () => {
  before(() => {
    cy.visit('/');
  });

  describe(
    // eslint-disable-next-line jest/valid-describe
    'homepage',
    {
      taskTimeout: 120000, // overrides default 60s, too little time to finish lighthouse tests
    },
    () => {
      it('should be fast', () => {
        cy.lighthouse();
      });
    },
  );
});

Cypress Version

7.7.0

Other

No response

[jennifer-shehane]

Are you using this plugin? https://www.npmjs.com/package/cypress-lighthouse

I've never used it before and it seems to error whenever I try to run it. And specifies requiring an older dependency of Cypress.

[Sybsw]

Hi @jennifer-shehane no I'm using https://www.npmjs.com/package/lighthouse v8.0.0. It adds the cy.lighthouse() command.

[metalix2]

Hi @jennifer-shehane

To add some more information We've setup a custom lighthouse plugin which is using lighthouse-ci. The way this interacts with cypress is by using the --remote-debugging-port and pointing the lighthouse test to the Cypress browser.

This was working for us as expected. Until we tried to upgrade to v7.6.0 where we see some changes in how the page with intercept mocking is effected. The page renders and then the mocks are applied.

Here are two different versions showing how the same page is loaded
V6
https://user-images.githubusercontent.com/2506718/128693920-e061ea2f-4bcc-4807-91a8-56431e01885a.mp4

V7
https://user-images.githubusercontent.com/2506718/128693931-da42703a-b690-4836-acd8-c9f18910b678.mp4

You can see in V7 that the No users is shown briefly before the mock finishes and then re-renders, causing the performance degradation.

Our assumption is that something has impacted how our mocking is done.

This is our mocking being set before the test

  before(() => {
    cy.mockUsers();
    cy.visit('/');
    cy.wait('@mockUsers');
  });

This is the mock being setup

Cypress.Commands.add('mockUsers', () => {
  cy.fixture('users.json').then((json) => {
    cy.intercept('GET', '/api/users?*', json).as('mockUsers');
  });
});

Anything which might help us would be greatly appreciated.

[metalix2]

Hi @jennifer-shehane Just wanted to check if this enough detail? Or any ideas why the mocking would be applied late in the render process?

[dapinto8]

Hi @metalix2, i'm experiencing lower performance using Lighthouse inside Cypress, i'm using this package https://www.npmjs.com/package/cypress-audit. Unfortunately i don't have previous results to campare, but i tried in multiple machines and enviroments, when i run Lighthouse in Chrome everything it's ok but when i run Lighthouse in Cypress the performance is worse. I tried with this repo https://github.com/dapinto8/cypress-test, it's just a next default app and has worse performance when runniung Lighthouse in Cypress.

Do you think this could be related with this issue?
I'm using Cypress 9.5.2

[dapinto8]

Hi @jennifer-shehane, I tried downgrading to 7.5.0 and Lighthouse performance result was the same as running Lighthouse with chrome (outside Cypress). I'm using a different package than @Sybsw, so it seems there's something that interfiers with Lighthouse since 7.6.0

You could try using this repository and changing the Cypress version. There is an spec with a command that runs Lighthouse.

[metalix2]

Hey @dapinto8 ,
Seems you've managed to answer your previous question. Your issue seems related to ours. One thing cypress states is that it isn't the ideal environment for performance tests. (https://docs.cypress.io/faq/questions/using-cypress-faq#Can-Cypress-be-used-for-performance-testing) Though we've seen a mix of success and failure from previous versions. If I had some real time to look into this. I'd like to identify the main change that happened between 7.5.0 and 7.6.0 It's not been a big priority for myself recently.

[metalix2]

I spent some more time on this again. Thought I'd share my learnings.

Assumption:

Cypress browser launcher is what is hindering lighthouse tests.

You can run lighthouse directly against the chrome browser that cypress will launch to run the tests (same debug-port defined in cypress) and the tests work as expected.

CYPRESS_REMOTE_DEBUGGING_PORT=33333 cypress open

If taking this approach and you run multiple lighthouse tests you should run the browser in incognito mode as it appears cypress browser caches the assets? e.g. first score is 85 then other scores in the high 90’s.

You can specify incognito like so.

on('before:browser:launch', (browser, launchOptions) => {

  launchOptions.args.push('--incognito');
}

You also are not able to mock or stub any network responses this way as the cypress runner controls the browsers network capabilities which we are bypassing. I believe this is what is responsible for the performance degradation.
When comparing between these two test methods you can see an increase in 500ms for every network call. This could be a network timeout or delay set somewhere.

Lighthouse test via remote_debugging browser and the other using cypress-browser

Reviewing the code changes between Version 7.5.0 and 7.6.0
I’ve reviewed the diff for the browsers in the server package but I can’t make out what exactly is the culprit to cause the slower tests.

Other Alternatives for mocking in lighthouse tests (Spoiler they don't work)

Puppeteer can launch browsers and manage network interceptions. See example below.

const browser = await puppeteer.launch({
  args: ['--disable-web-security'],
  ignoreHTTPSErrors: true,
  executablePath: '/Applications/Chromium.app/Contents/MacOS/Chromium',
  headless: false,
});

browser.on('targetchanged', async (target) => {
  const page = await target.page();
  await page.setRequestInterception(true);

  page.on('request', (request) => {
    if (request.url().startsWith('https://reqres.in/api/users')) {
      request.respond({
        status: 200,
        contentType: 'application/json',
        body: JSON.stringify(mockResponseObject),
      });
    } else request.continue();
  });
});

This would open a new browser and then on targetChanged event it would watch for requests and intercept the external request and stub it with the local fixture.

Though sadly this does not work due to lighthouse taking priority of the window/page. It needs to be intercepted at a browser level as Cypress does.

Outcomes

• Options can run tests in performant way without mocking by bypassing cypress browser.
• Could try cypress proxy to mock responses? (needs proving)

@ Cypress if there is anything else you need please don't hesitate to reach out.

[cypress-app-bot]

This issue has not had any activity in 180 days. Cypress evolves quickly and the reported behavior should be tested on the latest version of Cypress to verify the behavior is still occurring. It will be closed in 14 days if no updates are provided.

[cypress-app-bot]

This issue has been closed due to inactivity.