web.dev Lighthouse scores are different from DevTools scores

[samdutton]

When I run a Lighthouse audit from web.dev, I get different results than when running from the DevTools. Some of the web.dev Lighthouse warnings seem wrong, e.g. Does not register a service worker. I've tried Clear site data, etc.

---

web.dev

---

DevTools

---

DevTools settings

[ebidel]

DevTools is running an older version of LH (3.0.3), while web.dev is running the latest 3.2.1.

That said, if you're seeing major differences/issues with something like SW, try testing on new the
https://developers.google.com/speed/pagespeed/insights/ and see if there are similiar issues. That site only tests the performance category though.

Let us know!

[robdodson]

hm I saw Domenic report something similar about it not catching that sites had a service worker...

[samdutton]

Running from PSI:

From WebPageTest:

[robdodson]

@samdutton did you try here: https://developers.google.com/speed/pagespeed/insights/

I think web.dev is currently the only external site using the new version of lighthouse through insights.

[devnook]

It might be related to http issue too https://medium.com/dev-channel/web-dev-status-update-11-12-18-f9b42a693f65

[robdodson]

I think we can close this. Our scores are mostly mapping to what LH produces but they have a few known bugs on their end to fix http/2 and PWA.

GoogleChrome/lighthouse#6564
GoogleChrome/lighthouse#6566

[matteocargnelutti]

Hi @robdodson,

I am running into a similar issue, with the performance index.

Both https://web.dev/measure and https://developers.google.com/speed/pagespeed/insights/ return a performance index of ~70, while Lighthouse CLI and Lighthouse from DevTools return respectively 97 and 99.

I made sure DevTools' lighthouse was using the Simulated Fast 3G, 4X CPU Slowdown option.

Using the PageSpeed Insights API from the Google API Explorer Console, though, I also get a score of ~99.

What makes the performance score drop on https://web.dev/measure and https://developers.google.com/speed/pagespeed/insights/ is the presence of Google Tag Manager: removing it returns a performance score of 99 on all the Lighthouse version mentioned above.

Thank you in advance,

[robdodson]

Hey @matteocargnelutti,

If you leave Google Tag Manager on your page does DevTools still return 97/99 ?

[matteocargnelutti]

@robdodson Yes it does. Here's a recap.

With Google Tag Manager:

• web.dev: Performance ~70
• PageSpeed Insights: Performance ~70
• Lighthouse CLI: Performance ~97
• DevTools Lighthouse: Performance ~99
• PageSpeed Insights API: Performance ~99

Without Google Tag Manager:

• web.dev: Performance ~99
• PageSpeed Insights: Performance ~99
• Lighthouse CLI: Performance ~97
• DevTools Lighthouse: Performance ~99
• PageSpeed Insights API: Performance ~99

Thank you,

[paulirish]

"PageSpeed Insights API" is the exact same backend as "web.dev" and "PageSpeed Insights" so i'd expect its number to match those. Are you sure they differ?

[matteocargnelutti]

@paulirish

I just ran the tests again, just to be sure, with Google Tag Manager on:

• web.dev: Performance 72
• PageSpeed Insights: Performance 71
• PageSpeed Insights API: Performance 100 (see output below)
• Lighthouse CLI: Performance 96
• DevTools Lighthouse: Performance 99

PageSpeed Insights API output for performance (excerpt):

"categories": {
 "performance": {
   "id": "performance",
   "title": "Performance",
   "score": 1,
   "auditRefs": [ ... ]
 }
},

web.dev and PageSpeed Insight both indicate a "Time to Interactive" of 7.5s while DevTools' Lighthouse returns 1.7s for the same metric.

Thank you,

[paulirish]

@matteocargnelutti can you share your url? you can email me (my gh @ google do com) if you want.

[luisalvarado]

Try https://www.kidstherapyfinder.com it gives me very different results between wed.dev and Google Chrome devtools using the latest stable version of Google Chrome.

[BluePigMedia]

We are running into the same issue with significant delta between Google testing platforms for many sites we work on.

Example: https://davidc353.sg-host.com/

https://web.dev/measure/
100 Performance Score
FCP 0.3s
Speed Index 0.6s
LCP 1.2s
Time to Interactive 0.3s
Total Blocking Time 0 ms
Cumulative Layout Shift 0.01

Lighthouse in Chrome Dev Tools (Incognito Mode) on Mac Desktop Running Version 95.0.4638.69 (Official Build) (x86_64) with no extensions running

80 Performance Score
FCP 1.9s
Speed Index 1..9s
LCP 3.4s
Time to Interactive 4.3s
Total Blocking Time 370 ms
Cumulative Layout Shift 0.006

Page Speed Insights

58 Performance Score
FCP 1.9s
Speed Index 3.4s
LCP 3.0s
Time to Interactive 7.2s
Total Blocking Time 3490 ms
Cumulative Layout Shift 0.01

The reason this is important is Core Web Vital and Page Experience. One Google tools is rating our work as Gods, one as average student and one as a failure. Does anyone have any idea on what is going on and which of the three Google tools we should trust?

[NikkiMather]

I'm having the exact same issue. I assume the devs have, uh, just stopped caring?

[devnook]

Not exactly. Web.dev is using now Pagespeed Insights API, so the results should be the same as on PSI: https://pagespeed.web.dev/
You can read more about the differences in the FAQ at the bottom of the page: https://web.dev/measure/
If you still see discrepancies, please share the url you're measuring and I'll forward it to respective team.