Question - differences between web-vitals library, web-vitals extension, and CrUX report

[skaraman]

Hello again,

I'd like to explain that we are trying to use web-vitals library in a way so that we can get daily reports of metrics in our GA, in a similar way we see Monthly reports in our CrUX Dashboard v2.

We're confused about how CrUX/Page Speed Insights and the web-vitals extension (https://github.com/GoogleChrome/web-vitals-extension) report metrics, because it doesn't seem to be the same way as the web-vitals library. Are there differences in how these tools report metrics vs the library?

[ubertu]

I'm not sure to what you are referring.. I mean, are you referring about the values coming from different sources? they are not similar?

[psabharwal123]

@ubertu yes values from different sources are not similar. We are trying to replicate how chrome measures these metrics (since that will be important for SEO) by using this library.

[ubertu]

web-vitals library will return you a real and full value of the metrics, based on all the real user data. They are evaluated based on all the users the browser is able to calculate them. The others are somehow statistical measurement. Some time based on lab data and sometime based on sampling data. I would say the more accurate your measurement are, the better is, so definitivelly to me what you need is to use the library

[psabharwal123]

thanks for the reply, am i correct to understand that for SEO google will use the CrUX data which is also measured in lab?

[ubertu]

Who knows what google will do.. for sure they have the way to retrieve real data. Told that, using real data will make you in an ideal shape, misregard what they will.use

[psabharwal123]

It's not really a question of real data, both this library and CrUX provide real data, this is more of a question of how chrome measures the data, for a Google engineer.

[Zizzamia]

But, don't know if you are aware, but CrUX measure performance based on pathname, which means it mix results from LoggedIn and LoggedOut users. Also the CrUX dataset will always be a subset of your production field data, as not every user in Chrome gives permission to collect and report those metrics back to CrUX.

More to read, here.

So whatever you will measure in the field based on your production user will always diverge from CrUX. cc @rviscomi

Just make sure to use your Field Data as proxy to understand improvements and regression trends in your App. And have CrUX as ultimate validator of your work.

[psabharwal123]

The problem is our data shows lcp to be a lot less than from crux, most likely because we have a carousel, so I wanted to understand at what point does chrome send measurements. I understand data from field and crux would not be similar but I would expect some correlation.

On a side note we also measure mix of logged in and logged out users and ga data is also sampled.

[rviscomi]

Thanks for the cc, @Zizzamia. It's true that CrUX is a sample of the larger visitor population; it only includes Chrome users who opt-in to sharing usage statistics. Still, we've found the sampling is representative and there's a strong correlation between CrUX and other field data sources, so I wouldn't expect to see metrics vary too widely. The web vitals JS library powers the extension and it's built very similarly to how CrUX is measured by Chrome, so especially within those tools the data should be closely correlated.

@skaraman @psabharwal123 could you describe your site architecture and how you're measuring your own field data? It sounds like you're connecting the web vitals JS library to your GA account. I'd also be curious to hear if your site is an SPA, as that can complicate things.

[psabharwal123]

Sure, our site is a SPA and PWA. FID seems to have a good correlation but LCP and CLS do not. We measure vitals only for the first page load, we send cumulative measurement for each metric once user interacts with the page (scroll/click/touch/etc)

[rviscomi]

How much are the two data sources diverging, quantitatively? I'd be especially interested in the percent of "good" experiences and 75th percentiles for each CWV. And are you able to tell if this applies to all users or only certain segments, like mobile?

Being an SPA means that CrUX is only able to measure first page load experiences, but given that you're comparing it with first page load experiences from your own field data, it shouldn't be affecting the results.

[psabharwal123]

sorry for the late reply, they seem to diverge from about 5 to 20%. Our tracking is for all users, mobile desktop, guest, club.
i noticed #75 and #79 which might be affecing us, and #61 would be fixed in chrome 88 i believe which might help

[tunetheweb]

I believe this is now covered by this doc: https://web.dev/crux-and-rum-differences/