Add POC code to inject web vital js script

[ankur22]

Closes: #653

This POC tests to see how viable it is to inject Google's Web Vital JS script into each website that is navigated to.

Result

When testing the following test:

import { chromium } from 'k6/x/browser';
import { sleep } from 'k6';

export default function () {
  const browser = chromium.launch({
    headless: false,
  });
  const context = browser.newContext();
  context.addInitScript(`{
    function print(metric) {
      const m = {
        id: metric.id,
        name: metric.name,
        value: metric.value,
        rating: metric.rating,
        delta: metric.delta,
        numEntries: metric.entries.length,
        navigationType: metric.navigationType,
        url: window.location.href,
      }
      console.log('xk6-browser.web.vital.metric=' + JSON.stringify(m))
    }

    async function load() {
      let {
        onCLS, onFID, onLCP, onFCP, onINP, onTTFB
      } = await import('https://unpkg.com/web-vitals@3?module');

      onCLS(print);
      onFID(print);
      onLCP(print);
  
      onFCP(print);
      onINP(print);
      onTTFB(print);
    }

    load();
  }`);
  const page = context.newPage();

  page.goto('https://test.k6.io', { waitUntil: 'networkidle' })
  .then(() => {
    console.log("done")
  })
  .finally(()=> {
    sleep(6);
    page.close();
    browser.close();
  });
}

This is the output that we're interested in:

browser_cumulative_layout_shift..........: avg=0.000057 min=0.000057 med=0.000057 max=0.000057 p(90)=0.000057 p(95)=0.000057
browser_cumulative_layout_shift_good.....: 1      0.125174/s
browser_first_contentful_paint...........: avg=466.3ms  min=466.3ms  med=466.3ms  max=466.3ms  p(90)=466.3ms  p(95)=466.3ms 
browser_first_contentful_paint_good......: 1      0.125174/s
browser_first_input_delay................: avg=7.79ms   min=7.79ms   med=7.79ms   max=7.79ms   p(90)=7.79ms   p(95)=7.79ms  
browser_first_input_delay_good...........: 1      0.125174/s
browser_interaction_to_next_paint........: avg=8ms      min=8ms      med=8ms      max=8ms      p(90)=8ms      p(95)=8ms     
browser_interaction_to_next_paint_good...: 1      0.125174/s
browser_largest_content_paint............: avg=466.39ms min=466.39ms med=466.39ms max=466.39ms p(90)=466.39ms p(95)=466.39ms
browser_largest_content_paint_good.......: 1      0.125174/s
browser_time_to_first_byte...............: avg=326.7ms  min=326.7ms  med=326.7ms  max=326.7ms  p(90)=326.7ms  p(95)=326.7ms 
browser_time_to_first_byte_good..........: 1      0.125174/s

Analysis

The results are positive as we can retrieve the measurements for all of the Core Web Vitals, as well as three others which could be useful now or in the future. We should probably concentrate on the core Web Vitals and get a good understanding of those.

Some areas that need further investigation and input from product/users:

1. We probably need our own metric type, instead of using metrics.Trend.
   1. How do we display the rating in the summary in a concise way?
   2. We should concentrate on the 75th percentile as suggested by Google: "to ensure you're hitting the recommended target for most of your users, a good threshold to measure is the 75th percentile of page loads, segmented across mobile and desktop devices."
2. We need to tidy up and move this JS injection code within the code somewhere, possibly with the other injected script.
3. We probably don't want to aggregate the web vitals for all pages that are navigated, but instead split them up based on the frame and sub frame.
4. There's a limitation to this library in that it doesn't work for sub frames, so we would have to inject this library into the sub frame and display them separately. I don't think aggregating the root frames and the sub frames is going to be straight forward.
5. Can we use the Web Vitals with SPAs?
6. We will face the same problem we do with aggregated HTTP metrics - they're not all that useful until you split them out per page/group. But that obviously depends on the user to implement grouping.
   1. Grouping doesn't work with async APIs -- group doesn't work with async calls well  k6#2728.
7. We would likely need to bundle this script with the binary:
   1. Allows us to use this library in a locked down test environment that may not have access to all of the internet.
   2. Avoids XSS issues.
8. Grafana Faro also uses this.

Measurement Results And Comparison

Conclusion

I believe that this is likely to be the solution we go with, at least in the short term. It gives us what we need for now, we just need to fine tune it so that it's concise, simple to understand and all the information they need is displayed.

[inancgumus]

Nice research, thanks!

I haven't looked at the code deeply, and skimmed it, but it looks fine.

I believe that this is likely to be the solution we go with, at least in the short term. It gives us what we need for now, we just need to fine tune it so that it's concise, simple to understand and all the information they need is displayed.

Looks like so! 👍

Some areas that need further investigation and input from product/users:

1. We probably need our own metric type, instead of using metrics.Trend.

Yes, it might be better to wrap it in a higher type. Maybe even other metrics too.

1. How do we display the rating in the summary in a concise way?

I believe this is something we need to discuss with the product team (@markjmeier @dgzlopes) and @mdcruz. And then decide what the result should look like.

2. We need to tidy up and move this JS injection code within the code somewhere, possibly with the other injected script.

Yes, we can embed it as another script once for each frame.

3. We probably don't want to aggregate the web vitals for all pages that are navigated, but instead split them up based on the frame and sub frame.

When we send the metrics to k6, it will do grouping etc. for us. (Update: You explained later that it won't work with the async APIs 😮‍💨 ).

4. There's a limitation to this library in that it doesn't work for sub frames, so we would have to inject this library into the sub frame and display them separately. I don't think aggregating the root frames and the sub frames is going to be straight forward.

Since we'll be injecting the script for each frame, we can collect data from every frame?

5. Can we use the Web Vitals with SPAs?

🤷 We don't have a strong SPA support at the moment anyway :-]

7. We would likely need to bundle this script with the binary:

   1. Allows us to use this library in a locked down test environment that may not have access to all of the internet.
   2. Avoids XSS issues.

8. Grafana Faro also uses this.

👍

[ankur22]

Since we'll be injecting the script for each frame, we can collect data from every frame?

Yep, that's correct, so might not be such a big deal. We will need to think about how we can display it the summary though, or maybe we combine all the results from each frame for a single page. This will require some further investigation.

[markjmeier]

How do we display the rating in the summary in a concise way?

Meaning Good/Needs Improvement/Poor?

First and foremost, lets not overcomplicate the summary. Summary is just that - a summary. For k6 tests this is a test wide aggregation. Perhaps we take the same approach here - multiple page loads so an aggregation of vitals for them all:

First input delay avg: 150 ms - needs improvement

But maybe this isn't clear that there could be multiple pages - right?

What if the pattern was something like:

First Input Delay: 4/6 pages - Good Where 4 individual FID's are "good" and needs improvement or poor can be treated as failures?

thoughts? CC: @dgzlopes

[ankur22]

This was a POC, so closing it.