Investigate potential discrepancies in Web Vitals Metrics

[inancgumus]

In our recent pull requests, #943 and #949, we have resolved issue #914, which pertained to the inconsistent reporting of Web Vitals metrics. Following these updates, our focus now is to examine and verify if there are any disparities in the measurements provided by our tool compared to other tools.

To ensure the robustness and accuracy of our metrics, we aim to conduct a thorough investigation. This will help us to identify and implement any additional changes or fixes, should they be necessary. We welcome any insights or suggestions on this matter to aid in our investigation.

Possible tools to compare can be Lighthouse, Faro, etc.

[ka3de]

Here are the stats for Web Vitals metrics comparison between Google Lighthouse, Grafana Faro and k6 Browser (c959b57):

Site: grafana.com

Run 1	FCP	LCP	CLS
Lighthouse	800 ms	1.6 s	0.003
Faro	960 ms	1.28 s	0.0460 (*)
k6 Browser	288.19 ms	0.386 s	0.001493

Run 2	FCP	LCP	CLS
Lighthouse	800 ms	1.5 s	0.003
Faro	890 ms	1.10 s	0.0536 (*)
k6 Browser	292.5 ms	0.385 s	0.001554

Run 3	FCP	LCP	CLS
Lighthouse	700 ms	1.5 s	0.004
Faro	909 ms	1.09 s	0.0562 (*)
k6 Browser	292.4 ms	0.380 s	0.002158

(*) In contrast to Lighthouse and k6 Browser, Grafana Faro CLS metric corresponds to the aggregated values in a time interval measured from the instrumented application code itself, is not based on a single sample and is measured based on real user interaction, therefor I understand this discrepancy in CLS makes sense, as a real user interaction would produce more changes in the page than an automated test that just navigates to the page.

Site: test.k6.io

Run 1	FCP	LCP	CLS
Lighthouse	600 ms	600 ms	0
k6 Browser	556.1 ms	556.1 ms	0

Run 2	FCP	LCP	CLS
Lighthouse	500 ms	600 ms	0
k6 Browser	602.6 ms	602.6 s	0

Run 3	FCP	LCP	CLS
Lighthouse	600 ms	600 ms	0
k6 Browser	461.79 ms	461.79 ms	0

Grafana Faro was not used in this test, as it requires instrumentation in the web application code. Nevertheless, because it uses the same JS library in order to measure Web Vitals metrics as k6 Browser does, so we can assume that for a site as static as test.k6.io the metrics would have matched. See conclusions for a better explanation on this.

Conclusions:

What we can observe is that on sites that are very static (e.g.: test.k6.io) the Web Vitals metrics measured from the three tools are pretty much the same. The discrepancies are observed when testing more complex and dynamic sites (e.g.: grafana.com). In these cases, values reported from Faro and Lighthouse match more similarly than the ones reported by k6 Browser.

If we compare current k6 browser's main HEAD (c959b57) (which includes #949 and #943) with v0.10.0; v0.10.0 version does not report LCP metric consistently. In contrast, c959b57 version reports it (probably due to using reportAllChanges flag in web_vital_init.js script) but the measurements are clearly off compared to the other tools.

Therefor my understanding is that we are still missing Web Vitals metrics, probably due to a race condition between the metrics reporting/parsing/pushing and the iteration end.

ANNEX

These are the tests executed for k6, which consist on a single page navigation to the site under test:

c959b57 version

import { browser } from 'k6/x/browser';

export const options = {
  scenarios: {
    ui: {
      executor: 'shared-iterations',
      options: {
        browser: {
            type: 'chromium',
        },
      },
    },
  },
};
  

export default async function () {
  const page = browser.newPage();

  try {
    await page.goto('https://site.under.test.example.com')
  } finally {
    page.close();
  } 
}

v0.10.0 version

import { chromium } from 'k6/experimental/browser';

export default async function () {
  const browser = chromium.launch();
  const page = browser.newPage();

  try {
    await page.goto('https://site.under.test.example.com')
  } finally {
    page.close();
    browser.close();
  } 
}

[ka3de]

Closing, as investigation is already done and associated issues (#960) have been created.