New Core Web Vitals tech report

[sarahfossheim]

Some of us are currently migrating the Core Web Vitals report (cwvtech.report) off Data Studio into the HTTPArchive website. We’ll maintain feature/data parity (but with an updated interface), and are also looking at possible future additional data or ways of interacting with it.

We’d love to learn more about how you use the current report, what works well for you and what doesn’t, and what you’d like to see added/changed/fixed.

These are the things we’d like input on, feel free to either respond in a comment, create a new issue, or contribute on Slack ⬇️

Core Web Vitals data & insights

We want to make sure we present data that is useful, so for that we’d love to get a bit of background on what kind of (core web vitals) insights you’re after, and how you’d use those.

Think of: What CWV related information are you after? What kind of values and metrics are you interested in? How and why do you use those? What do they tell you? Are there specific values or patterns you’re on the lookout for (eg. rising/declining values, combination of metrics), and why? What other info are you after?

The current report

How is the current report used, what works well, what doesn’t work well at all, and what’s missing?

Think of: What kind of insights are you after in the cwvtech.report? Do you know beforehand or decide based on the data what you’re interested in? How do you use the report? Do you set any filters to find the information you’re after, and if so, which ones and why? Do you mainly use drilldown or compare, or both? What works well with the current tool? Is something missing in the report?

Sketches

These are sketches from earlier this spring. Please not that the charts have placeholder styling (so they won’t be black/white in reality) and also placeholder data, as we’re in first instance looking for feedback on the functionality, structure, filters, and chart types.

Sketch 1:

The same graphs and filters as in the cwvtech.report, but ported over inside the HTTPArchive’s website (similar to the other reports). This is what a first feature parity without much else added could look like. Both the drilldown and comparison view can be filtered on rank, geography, origins, and be filtered further by date ranges. There's data on the different core web vitals, lighthouse scores, and page composition, all displayed over time and broken down by client. Multi-tech shows the data for the different selected technologies over time, either for mobile or desktop.

Sketch 2

A possible future version of the report, with additional metrics and graphs (such as a breakdown of the median CWVs, lighthouse audits, etc). At this point the report can be broken into several sections, for each of the big areas (CWVs, Lighthouse, etc).

We’d love to hear what you think 😊

[kevinfarrugia]

Sketches and overall idea look very promising. 🙌

My most common use-case for using cwvtech.report is to compare different technologies using real world data. In the past I have reached out to third parties to flag a performance issue and used the cwvtech.report to illustrate the third party with similar libraries, but this would also require a manual query to extract the cause of the issue. For example, MyFavoriteTracker was invalidating bfcache by using an unload listener on 100,000 sites. So it is super interesting to be able to get the lighthouse audits (sketch 2, image 4) for each technology as this would remove the requirement for running a query while making the data more accessible.

I look forward to the option to be able to filter by rank, to include distributions, and to get the median (or P75) value for each CWV.

I do not think it is covered in the sketches, but a small suggestion to allow adoption to be expressed as a percentage of all sites in that data set. For example, if we are looking at Wordpress, filtered by rank: 10,000, then it would be nice to easily understand that 60% (example) of sites are using Wordpress. Currently it is not possible or requires some manual computation since the total number of sites is not visible in the report.

Apart form the distribution for the last month, it would be nice if we could see the distribution over time (similar to what treo.sh does):

Hope that helps. 😁

[Tiggerito]

Is it possible to optionally limit the reports to a specific device type?

In my distribution reports, I include a pie chart to clarify how the distribution is around the 75th percentile. Alternatively, the 75th percentile could be marked on the distribution chart (I could not work out how to do it in Looker).

[sarahfossheim]

Thanks for the feedback! Several people have mentioned the P75 value, so we're looking into adding it 👍🏻

Is it possible to optionally limit the reports to a specific device type?

Do you mean by mobile/desktop? If so, yes! In the current cwvtech.report there already is a client dropdown in the comparison page. In the new report, both the drilldown and comparison page will be filterable by device type.

Screenshots from the client selection in the new dashboard (in development):

For most components, only the selected client will be shown.

The timeseries in the drilldown page (one tech) show both device types simultaneously.

The timeseries are filterable by client as well, directly by interacting with the chart using Highchart's built-in functionality. The legend can be used to toggle different categories on the chart on and off.

Note that all of these screenshots are from a work in progress / development version, things may change :)

[pedddro]

I think one way to improve would be to implement a way to visualize the distribution in the number of different website origins to a certain technology, that way we could notice outliers. Not sure if filtering outliers would be feasible to implement but it's something I think would be useful.