Tech Report: overall good CWVs is misleading

[maiieul]

Hi, Qwik core team member here 👋

Looking at the reports for Qwik and Astro, I observe that the "overall good CWVs" results are a lot lower than the average of all metrics combined, especially for Qwik.

For example the average of FCP, LCP, CLS, INP, TTFB for Astro on mobile is ~73, while it's overall good CWVs on mobile is 63. For Qwik it's even worse: average on mobile is 83, while overall good is only 57.

This might be explainable by the fact that the Qwik dataset is smaller and more apps have just one metric not passing than if it were on a bigger dataset, but I still find it quite misleading. People might be looking at Qwik and think "well this is not so good.." (especially because this is the first view that they'll see), when in reality it often reaches high scores, demonstrated by the good total average score.

If my assumption is correct that this is due to the smaller dataset, I'd like to propose a new option to select: "Average CWVs score". And rename the "overall good CWVs" into "Passes CWVs", which I would show as the second selectable option.

Data from Aug 1, 2025: https://httparchive.org/reports/techreport/tech?good-cwv-over-time=overall&tech=Qwik%2CAstro&geo=ALL&rank=ALL&page=1&client=mobile

[tunetheweb]

Yes there's lies, damn lies, and statistics.

"Correlation is not causation" as people are prone to say to excuse themselves just as as their about to imply a correlation.

Some technologies are more likely to be used by more complex sites. Some are used on many more legacy sites, that haven't benefited from the latest web platform enhancements to make their sites faster.

Being used on less sites has advantages and disadvantages. On the one side it can be argued this is new tech, and arguably used by sites that are more performance focused. On the other there is high variability due to being a smaller dataset.

In this case CWV pass rates are based on the CWVs only (LCP, CLS, and INP - not FCP and TTFB as you've stated) and drilling down into those it looks like:

• LCP - Qwik out performs Astro in this dataset.
• INP - Qwik looks to be slightly under performing Astro in this dataset.
• CLS - is highly variable for Qwik sites and this looks to be a bad month for CLS and is likely what's dragging this down as you were out performing them last month.

If my assumption is correct that this is due to the smaller dataset, I'd like to propose a new option to select: "Average CWVs score". And rename the "overall good CWVs" into "Passes CWVs", which I would show as the second selectable option.

I'm not sure how you mean to measure this since CWV is based on three metrics not one. Average across three? That sounds like taking average of averages and getting to the point where the result is meaningless!

The "passes CWV" is a very basic measure to be sure, but I'm not sure some calculated one would be any better. Or that we wouldn't have Astro (or some other technology that fares better in the current status quo) on here next month asking us to switch back.

[maiieul]

In this case CWV pass rates are based on the CWVs only (LCP, CLS, and INP - not FCP and TTFB as you've stated)

Yes, sorry for the amalgam.

---

The confusion originally came from the old report, where it was not clear that "good CWVs" are in fact the percentage of apps that pass the CWVs. Me and my friends thought it was the average of all metrics.

It's clearer in the new report with the explanation and title, but I would still rename the selectable option to "Passes CWVs":

---

I'm not sure how you mean to measure this since CWV is based on three metrics not one. Average across three? That sounds like taking average of averages and getting to the point where the result is meaningless!
Maybe.

To me it makes sense to show the average of averages. It gives people a glimpse into what kind of CWVs scores they should expect with a given technology. The cons of showing the average could be that (1) a technology could be very good at 2 metrics and bad at one, which would mean the CWVs never pass, and (2) one more option to add to the select input. I find those cons acceptable.

By choosing to show "passes CWVs" to be the first in view, projects with smaller datasets will be impacted with overly bad (or good? to be verified..) results, misleading those who are looking at the dataset. Perhaps simply adding a note saying that small datasets can easily show an inaccurate picture would go a long way? I personally assumed that 1200 origins would be statistically significant enough.

[tunetheweb]

I'm not sure how you mean to measure this since CWV is based on three metrics not one. Average across three? That sounds like taking average of averages and getting to the point where the result is meaningless!
Maybe.

To me it makes sense to show the average of averages. It gives people a glimpse into what kind of CWVs scores they should expect with a given technology. The cons of showing the average could be that (1) a technology could be very good at 2 metrics and bad at one, which would mean the CWVs never pass, and (2) one more option to add to the select input. I find those cons acceptable.

Since the metrics are different for each (LCP is measures in seconds, INP in milliseconds and CLS in fractions) averaging them doesn't make sense. Unless you mean average number of passing sites averaged across all three metrics? I'm not loving this new definition of "Core Web Vitals pass rate" that we've not used else where.

By choosing to show "passes CWVs" to be the first in view, projects with smaller datasets will be impacted with overly bad (or good? to be verified..) results, misleading those who are looking at the dataset. Perhaps simply adding a note saying that small datasets can easily show an inaccurate picture would go a long way? I personally assumed that 1200 origins would be statistically significant enough.

Other than a larger degree of variability, I don't understand why you think a smaller dataset leads to bad (or good) results? is the expectation that the law of large numbers leads to a more accurate representation of that technology? As I said in my first reply, I'm not convinced that, even if that were true, that would allow cross-measurement under the assumption that technologies are equal as there are inherit biases anyway (e.g. those using Qwik are likely more performance focused, those using React are more likely building more complex interactive sites than say WordPress sites...etc). The numbers are presented as is and it's up to the reader to interpret them. I don't think we should get into the habit of saying "these numbers are super accurate, these are likely suspect".

[maiieul]

Unless you mean average number of passing sites averaged across all three metrics?
Yes that's what I meant. Now that you put it like that it's indeed not a very tangible measurement compared to the others.

It might just a UI issue here in the end. All metrics in this view are reprensentations of good CWVs. To make things more obvious, I would rename the options in the select to:

• Passes CWVs
• Good LCP
• Good CLS
• Good FCP
• Good TTFB
• Good INP

This way there's less room for confusion. The new website is already a lot clearer than the old on that regard, but I think this would be a nice little UI improvement.

But this gets me thinking, why do we need a "Passes CWVs" view to begin with? Unless google gives a bonus to sites that pass all 3 CWVs, it doesn't seem to be that meaningful either. If it were me, I'd show the 5 metrics graphs in one view (LCP, CLS, INP at the top, FCP, TTFB at the bottom) and remove the "passes CWVs" graph. That should fit without looking bad and would remove any confusion. And for the summary section, I'd put LCP, CLS and INP similar to A11Y and SEO, in place of the "good CWVs" column with the progress bar.

[tunetheweb]

Yeah you're not wrong that "Passes CWV" is a bit of an arbitrary metric. The only other place that really exposes this is PageSpeed Insights and some have argued that it shouldn't. Though I do make note of it in the monthly CrUX releases too: https://developer.chrome.com/docs/crux/release-notes. Overall it's a high-level target to aim for and the only meaningful way to quickly show one figure for the three different metrics for a quick summary.

I would rename the options in the select to:
... Good XXX

Happy to do that.

I'd show the 5 metrics graphs in one view (LCP, CLS, INP at the top, FCP, TTFB at the bottom) and remove the "passes CWVs" graph. That should fit without looking bad and would remove any confusion. And for the summary section, I'd put LCP, CLS and INP similar to A11Y and SEO, in place of the "good CWVs" column with the progress bar.

I'm against this. While that might work for a single technology, with multiple technologies in the same graph that would get very messy, very quickly.

[maiieul]

Happy to do that.

Great, thanks!

Overall it's a high-level target to aim for and the only meaningful way to quickly show one figure for the three different metrics for a quick summary.

Yeah I'd argue it's not helpful. In pagespeed, The 5 metrics are shown in one view, so it doesn't add anything really meaningful. This is why I suggest doing the same (5 in view) with graphs on the techreport.

I'm against this. While that might work for a single technology, with multiple technologies in the same graph that would get very messy, very quickly.

Even with only one graph in view it gets very messy above 5 technologies. Imo it's actually cleaner to show the 5 graphs and remove the "passes CWVs" one, which mostly adds noise since it doesn't mean much.

In fact you had two graphs next to one another in the previous report

Unless the two graphs have nothing to do with one another, I don't see this as bad UI/UX. In our case here, LCP, CLS and INP are part of the CWVs so it makes sense to show them at the same time for an easy overview between the 3.

If 3 on the same row is too much, you can do

LCP | CLS
   INP
FCP | TTFB

You could always add a "zoom" button to focus on one view at a time.

Up to you guys ofc. Just sharing my feedback ✌️