Performance numbers are on the extreme ends of the Prism when used Lighthouse chrome browser Plugin versus Cypress Audit library

[sarathk-blink]

When I run the Perf testing for google page using the chrome lighthouse plugin the numbers are 100s and 90s. I do the same using the cypress-audit library using cy.lighthouse the numbers are like 40s and 50s.

Same goes with the other pages that Im currently working on my project. I was wondering as to why there's such a huge gap.

[mfrachet]

It may (and will) vary depending:

• the environment you're running the lighthouse check
• the way you run lighthouse (from chrome vs from the CLI vs an online tool vs this module)
• the status of your build (production vs development)

And probably many other things that I don't have in mind. It's important to remember that this module uses the lighthouse node module with Cypress. Cypress is not free for the machine and it's possible that the lighthouse module may be different from the one ship in the browser.

I honestly don't have a strong answer to this question. The only thing I know is that there is a lot to consider in order to make "an accurate" difference (which again, might not be accurate because lighthouse can't be 100% deterministic).

[rad-der]

Hi,
I observe the same situation and here is what I found:

LH CI:
lighthouse https://xxxxx --view --form-factor=desktop --screenEmulation.disabled --only-categories=performance

cypress-audit:

   const desktopConfig = {
      formFactor: 'desktop',
      screenEmulation: { disabled: true },
      onlyCategories: ['performance']
    };
        cy.lighthouse(customThresholds, desktopConfig);

In both cases, the final performance score will be the same (LH CI = 40, cypress-audit = 40)

but when I added a --preset=desktop flag
lighthouse https://xxxxx --view --form-factor=desktop --screenEmulation.disabled --only-categories=performance --preset=desktop

LH CI perf score will be 95 -> the same result is visible also in lighthouse chrome dev tool

the same change for cypress-audit:

    const desktopConfig = {
      formFactor: 'desktop',
      screenEmulation: { disabled: true },
      onlyCategories: ['performance'],
      preset: 'desktop'
    };

performance score will be still 40.

Something is wrong with preset flag?
From LH readme:

--preset                       Use a built-in configuration.
                                 WARNING: If the --config-path flag is provided, this preset will be ignored.  [string] [choices: "perf", "experimental", "desktop"]

maybe you are using somewhere in your project --config-path and --preset was overridden?

[mfrachet]

Thank you so much for the help you provided @rad-der 🙏🏻 . This is awesome 👍🏻

[maciejtrzcinski]

I've found bypass solution, it's not ideal but works.
I've used the lighthouse default desktop settings.

screenEmulation:

const DESKTOP_EMULATION_METRICS = {  
  mobile: false,  
  width: 1350,  
  height: 940,  
  deviceScaleFactor: 1,  
  disabled: false,
};

throttling:

desktopDense4G: {  
  rttMs: 40,    
  throughputKbps: 10 * 1024,    
  cpuSlowdownMultiplier: 1,    
  requestLatencyMs: 0, // 0 means unset    
  downloadThroughputKbps: 0,    
  uploadThroughputKbps: 0,
}

    const customThresholds = {
      performance: 90,
    }

    const desktopConfig = {
      formFactor: 'desktop',
      screenEmulation: {
        width: 1350,
        height: 940,
        deviceScaleRatio: 1,
        mobile: false,
        disable: false,
      },
      throttling: {
        rttMs: 40,
        throughputKbps: 11024,
        cpuSlowdownMultiplier: 1,
        requestLatencyMs: 0,
        downloadThroughputKbps: 0,
        uploadThroughputKbps: 0,
      },
    }

    cy.lighthouse(customThresholds, desktopConfig)

[mfrachet]

Hello, quick follow-up on this issue. I have added a section in the LH docs to suggest using the settings provided by @maciejtrzcinski => https://github.com/mfrachet/cypress-audit/blob/master/docs/lighthouse.md#lighthouse-scores-may-be-different-between-local-run-and-cypress-audit

I would like to take some time with this before making them as default inside the library. If we see again this kind of issues, I'll put them as default and iterate from that :).

Thanks again everyone for this 🙏🏻 . Closing for now 😊

[bhavm24]

@mfrachet - link seems to be broken, could you please update