From 6db0337de4caf22fad75fee90c33c09a776a0147 Mon Sep 17 00:00:00 2001 From: Philip Walton Date: Mon, 1 Nov 2021 18:56:04 -0700 Subject: [PATCH 01/11] Add initial responsiveness metric draft --- .../content/en/blog/responsiveness/index.md | 301 ++++++++++++++++++ 1 file changed, 301 insertions(+) create mode 100644 src/site/content/en/blog/responsiveness/index.md diff --git a/src/site/content/en/blog/responsiveness/index.md b/src/site/content/en/blog/responsiveness/index.md new file mode 100644 index 00000000000..e2846484ef1 --- /dev/null +++ b/src/site/content/en/blog/responsiveness/index.md @@ -0,0 +1,301 @@ +--- +title: 'Feedback wanted: an experimental responsiveness metric' +subhead: An update on our plans for measuring responsiveness on the web. +description: An update on our plans for measuring responsiveness on the web. +authors: + - hbsong +date: 2021-11-03 +hero: image/eqprBhZUGfb8WYnumQ9ljAxRrA72/GNWyQZ5l1uy1b7s5ursN.jpeg +alt: Water droplet rippling outward +tags: + - blog # blog is a required tag for the article to show up in the blog. + - performance + - web-vitals +--- + +Earlier this year, the Chrome Speed Metrics Team shared [some of the +ideas](/better-responsiveness-metric/) we were considering for a +new responsiveness metric. We want to design a metric that better captures the +end-to-end latency of individual events and offers a more holistic picture of +the overall responsiveness of a page throughout its lifetime. + +We've made a lot of progress on this metric in the last few months, and we +wanted to share an update on how we plan to measure interaction latency as well +as introduce a few specific aggregation options we're considering to quantify +the overall responsiveness of a web page. + +We'd love to get [feedback](#feedback) from developers and site owners as to +which of these options would be most representative of the overall input +responsiveness of their pages. + +## Measure interaction latency + +As a review, the [First Input Delay (FID)](/fid/) metric captures the +[delay](/fid/#fid-in-detail) portion of input latency. That is, the time between +when the user interacts with the page to the time when the event handlers are +able to run. + +With this new metric we plan to expand that to capture the [full event +duration](/better-responsiveness-metric/#capture-the-full-event-duration), from +initial user input until the next frame is painted after all the event handlers +have run. + +We also plan to measure +[interactions](/better-responsiveness-metric/#group-events-into-interactions) +rather than individual events. Interactions are groups of events that are +dispatched as part of the same, logical user gesture (for example: +`pointerdown`, `click`, `pointerup`). + +To measure the total interaction latency from a group of individual event +durations, we are considering two potential approaches: + +- **Maximum event duration:** the interaction latency is equal to the largest + single event duration from any event in the interaction group. +- **Total event duration:** the interaction latency is the sum of all event + durations, ignoring any overlap. + +As an example, the diagram below shows a _key press_ interaction that consists +of a `keydown` and a `keyup` event. In this example there is a duration overlap +between these two events. To measure the latency of the key press interaction, +we could use `max(keydown duration, keyup duration)` or `sum(keydown duration, +keyup duration) - duration overlap`: + +{% Img src="image/eqprBhZUGfb8WYnumQ9ljAxRrA72/F55OBEhX6VvNEApTjldU.svg", +alt="ALT_TEXT_HERE", width="800", height="424" %} + +There are pros and cons of each approach, and we'd like to collect more data and +[feedback](#feedback) before finalizing a latency definition. + +{% Aside %} + The event duration is meant to be the time from the event hardware timestamp + to the time when the next paint is performed after the event is handled. But + if the event doesn't cause any update, the duration will be the time from + event hardware timestamp to the time when we are sure it will not cause any + update. +{% endAside %} + +{% Aside %} + For keyboard interactions, we usually measure the `keydown` and `keyup`. But + for IME, such as input methods for Chinese and Japanese, we measure the input + events between a `compositionstart` and a `compositionend`. +{% endAside %} + +## Aggregate all interactions per page + +Once we're able to measure the end-to-end latency of all interactions, the next +step is to define an aggregate score for a page visit, which may contain more +than one interaction. + +After exploring a number of options, we've narrowed our choices down to the +strategies outlined in the following section, each of which we're currently +collecting real-user data on in Chrome. We plan to publish the results of our +findings once we've had time to collect sufficient data, but we're also looking +for direct [feedback](#feedback) from site owners as to which strategy would +most accurately reflect the interaction patterns on their pages. + +### Aggregation strategies options + +To help explain each of the following strategies, consider an example page visit +that consists of four interactions: + +
+ + + + + + + + + + + + + + + + + + + + + +
InteractionLatency
Click120 ms
Click20 ms
Key press60 ms
Key press80 ms
+
+ +#### Worst interaction latency + +The largest, individual interaction latency that occurred on a page. Given the +example interactions listed above, the worst interaction latency would be 120 +ms. + +#### Budgets strategies + +[User experience +research](https://www.tactuallabs.com/papers/howMuchFasterIsFastEnoughCHI15.pdf) +suggests that users may not perceive latencies below certain thresholds as +negative. Based on this research we're considering several budget strategies +using on the following thresholds for each event type: + +
+ + + + + + + + + + + + + + + + + +
Interaction typeBudget threshold
Click/tap100 ms
Drag100 ms
Keyboard50 ms
+
+ +Each of these strategies will only consider the latency that is more than the +budget threshold per interaction. Using the example page visit above, the +over-budget amounts would be as follows: + +
+ + + + + + + + + + + + + + + + + + + + + + + + + + +
InteractionLatencyLatency over budget
Click + 120 ms20 ms
Click + 20 ms0 ms
Key press + 60 ms10 ms
Key press + 80 ms30 ms
+
+ +#### Worst interaction latency over budget + +The largest single interaction latency over budget. Using the above example, the +score would be `max(20, 0, 10, 30) = 30 ms`. + +#### Total interaction latency over budget + +The sum of all interaction latencies over budget. Using the above example, the +score would be `(20 + 0 + 10 + 30) = 60 ms`. + +#### Average interaction latency over budget + +The total over-budget interaction latency divided by the total number of +interactions. Using the above example, the score would be `(20 + 0 + 10 + 30) / +4 = 15 ms`. + +#### High quantile approximation + +As an alternative to computing the largest interaction latency over budget, we +also considered using a high quantile approximation, which should be fairer to +web pages that have a lot of interactions and may be more likely to have large +outliers. We've identified two potential high-quantile approximation strategies +we like: + +- **Option 1:** keep track of the largest interaction over budget. After every + 50 interactions, discard the largest and replace it with the next largest. +- **Option 2:** compute the largest 10 interactions over budget and choose a + value from that list depending on the total number of interactions. Given N + total interactions, select the (N / 50 + 1)th largest value, or the 10th value + for pages with more than 500 interactions. + +## Measure these options in JavaScript + +The following code example can be used to determine the values of the first +three strategies presented above. Note that it's not yet possible to measure the +total number of interactions on a page in JavaScript, so this example doesn't +include the average interaction over budget strategy or the high +quantile approximation strategies. + +```js +const interactionMap = new Map(); + +let worstLatency = 0; +let worstLatencyOverBudget = 0; +let totalLatencyOverBudget = 0; + +new PerformanceObserver((entries) => { + for (const entry of entries.getEntries()) { + // Ignore entries without an interaction ID. + if (entry.interactionId > 0) { + // Get the interaction for this entry, or create one if it doesn't exist. + let interaction = interactionMap.get(entry.interactionId); + if (!interaction) { + interaction = {entries: []}; + interactionMap.set(entry.interactionId, interaction); + } + interaction.entries.push(entry); + + const latency = Math.max(entry.duration, interaction.latency || 0); + worstLatency = Math.max(worstLatency, latency); + + const budget = entry.name.includes('key') ? 50 : 100; + const latencyOverBudget = latency - budget; + worstLatencyOverBudget = + Math.max(latencyOverBudget, worstLatencyOverBudget); + + // If this event adds additional latency, update the total over budget. + const newLatency = latency - (interaction.latency || 0); + if (newLatency > 0) { + totalLatencyOverBudget += newLatency; + } + + // Set the latency on the interaction so future events can reference. + interaction.latency = latency; + + // Log the updated metric values. + console.log({ + worstLatency, + worstLatencyOverBudget, + totalLatencyOverBudget, + }); + } + } +}).observe({type: 'event', buffered: true, durationThreshold: 50}); +``` + +{% Aside 'caution' %} + There are currently [a few + bugs](https://bugs.chromium.org/p/chromium/issues/list?q=label:responsiveness-bugs) + in Chrome that affect accuracy of the reported interaction timestamps. We are + working to fix these bugs as soon as possible, and we recommend developers + test these strategies in Chrome Canary to get the most accurate results. +{% endAside %} + +## Feedback + +We want to encourage developers to try out these new layout shift metrics on +their sites, and let us know if you discover any issue. + +Please also email any general feedback on the approaches outlined here to the +[web-vitals-feedback](https://groups.google.com/g/web-vitals-feedback) Google +group with "[Responsiveness Metrics]" in the subject line. We're really looking +forward to hearing what you think! From d1194e4013ab8a40a8d626c9db7414d0c70494b9 Mon Sep 17 00:00:00 2001 From: Philip Walton Date: Mon, 1 Nov 2021 20:14:36 -0700 Subject: [PATCH 02/11] Add alt text to the image --- src/site/content/en/blog/responsiveness/index.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/src/site/content/en/blog/responsiveness/index.md b/src/site/content/en/blog/responsiveness/index.md index e2846484ef1..f68478cc902 100644 --- a/src/site/content/en/blog/responsiveness/index.md +++ b/src/site/content/en/blog/responsiveness/index.md @@ -60,8 +60,9 @@ between these two events. To measure the latency of the key press interaction, we could use `max(keydown duration, keyup duration)` or `sum(keydown duration, keyup duration) - duration overlap`: -{% Img src="image/eqprBhZUGfb8WYnumQ9ljAxRrA72/F55OBEhX6VvNEApTjldU.svg", -alt="ALT_TEXT_HERE", width="800", height="424" %} +{% Img src="image/eqprBhZUGfb8WYnumQ9ljAxRrA72/F55OBEhX6VvNEApTjldU.svg", alt="A +diagram showing interaction latency based on event durations", width="800", +height="424" %} There are pros and cons of each approach, and we'd like to collect more data and [feedback](#feedback) before finalizing a latency definition. From 30d0257e7c39816fd30fb947555d3b191ed0fa84 Mon Sep 17 00:00:00 2001 From: Jeremy Wagner Date: Tue, 2 Nov 2021 09:35:07 -0500 Subject: [PATCH 03/11] Makes minor changes. - Changes a link's linked text. - Updates the diagram with a cleaner version. --- src/site/content/en/blog/responsiveness/index.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/src/site/content/en/blog/responsiveness/index.md b/src/site/content/en/blog/responsiveness/index.md index f68478cc902..ae9d630db06 100644 --- a/src/site/content/en/blog/responsiveness/index.md +++ b/src/site/content/en/blog/responsiveness/index.md @@ -30,8 +30,8 @@ responsiveness of their pages. ## Measure interaction latency -As a review, the [First Input Delay (FID)](/fid/) metric captures the -[delay](/fid/#fid-in-detail) portion of input latency. That is, the time between +As a review, the [First Input Delay (FID) metric captures the +delay portion](/fid/#fid-in-detail) of input latency. That is, the time between when the user interacts with the page to the time when the event handlers are able to run. @@ -60,7 +60,7 @@ between these two events. To measure the latency of the key press interaction, we could use `max(keydown duration, keyup duration)` or `sum(keydown duration, keyup duration) - duration overlap`: -{% Img src="image/eqprBhZUGfb8WYnumQ9ljAxRrA72/F55OBEhX6VvNEApTjldU.svg", alt="A +{% Img src="image/jL3OLOhcWUQDnR4XjewLBx4e3PC3/td8jMQBTjyR34ZO1QBvp.svg", alt="A diagram showing interaction latency based on event durations", width="800", height="424" %} From 4770868c496db1797a74debe4a2b295fe502ddb8 Mon Sep 17 00:00:00 2001 From: Philip Walton Date: Tue, 2 Nov 2021 08:51:41 -0700 Subject: [PATCH 04/11] Update src/site/content/en/blog/responsiveness/index.md Co-authored-by: Milica --- src/site/content/en/blog/responsiveness/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/site/content/en/blog/responsiveness/index.md b/src/site/content/en/blog/responsiveness/index.md index ae9d630db06..cf24f7ca0d0 100644 --- a/src/site/content/en/blog/responsiveness/index.md +++ b/src/site/content/en/blog/responsiveness/index.md @@ -1,5 +1,5 @@ --- -title: 'Feedback wanted: an experimental responsiveness metric' +title: 'Feedback wanted: An experimental responsiveness metric' subhead: An update on our plans for measuring responsiveness on the web. description: An update on our plans for measuring responsiveness on the web. authors: From 228c481740eab8db64e4f971d1c353b6ab6802d9 Mon Sep 17 00:00:00 2001 From: Philip Walton Date: Tue, 2 Nov 2021 08:51:48 -0700 Subject: [PATCH 05/11] Update src/site/content/en/blog/responsiveness/index.md Co-authored-by: Milica --- src/site/content/en/blog/responsiveness/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/site/content/en/blog/responsiveness/index.md b/src/site/content/en/blog/responsiveness/index.md index cf24f7ca0d0..5e97b74b6c4 100644 --- a/src/site/content/en/blog/responsiveness/index.md +++ b/src/site/content/en/blog/responsiveness/index.md @@ -223,7 +223,7 @@ we like: - **Option 1:** keep track of the largest interaction over budget. After every 50 interactions, discard the largest and replace it with the next largest. -- **Option 2:** compute the largest 10 interactions over budget and choose a +- **Option 2:** Compute the largest 10 interactions over budget and choose a value from that list depending on the total number of interactions. Given N total interactions, select the (N / 50 + 1)th largest value, or the 10th value for pages with more than 500 interactions. From 322cd5b9cabeebd0a7480c3bb70dacbf9474b7ae Mon Sep 17 00:00:00 2001 From: Philip Walton Date: Tue, 2 Nov 2021 08:52:52 -0700 Subject: [PATCH 06/11] Update src/site/content/en/blog/responsiveness/index.md Co-authored-by: Milica --- src/site/content/en/blog/responsiveness/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/site/content/en/blog/responsiveness/index.md b/src/site/content/en/blog/responsiveness/index.md index 5e97b74b6c4..3795bc7f2b3 100644 --- a/src/site/content/en/blog/responsiveness/index.md +++ b/src/site/content/en/blog/responsiveness/index.md @@ -221,7 +221,7 @@ web pages that have a lot of interactions and may be more likely to have large outliers. We've identified two potential high-quantile approximation strategies we like: -- **Option 1:** keep track of the largest interaction over budget. After every +- **Option 1:** Keep track of the largest interaction over budget. After every 50 interactions, discard the largest and replace it with the next largest. - **Option 2:** Compute the largest 10 interactions over budget and choose a value from that list depending on the total number of interactions. Given N From 6943652419fd0eb162e1f34dec01e95662d61ba9 Mon Sep 17 00:00:00 2001 From: Philip Walton Date: Tue, 2 Nov 2021 08:58:28 -0700 Subject: [PATCH 07/11] Update src/site/content/en/blog/responsiveness/index.md Co-authored-by: Milica --- src/site/content/en/blog/responsiveness/index.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/src/site/content/en/blog/responsiveness/index.md b/src/site/content/en/blog/responsiveness/index.md index 3795bc7f2b3..b3d3c1b6c72 100644 --- a/src/site/content/en/blog/responsiveness/index.md +++ b/src/site/content/en/blog/responsiveness/index.md @@ -103,23 +103,23 @@ that consists of four interactions: - + - + - + - + - +
InteractionLatencyLatency
Click120 ms120 ms
Click20 ms20 ms
Key press60 ms60 ms
Key press80 ms80 ms
From 9d2035217d53a7c09132e06adf11fa86dd3a343b Mon Sep 17 00:00:00 2001 From: Philip Walton Date: Tue, 2 Nov 2021 08:58:35 -0700 Subject: [PATCH 08/11] Update src/site/content/en/blog/responsiveness/index.md Co-authored-by: Milica --- .../content/en/blog/responsiveness/index.md | 20 +++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/src/site/content/en/blog/responsiveness/index.md b/src/site/content/en/blog/responsiveness/index.md index b3d3c1b6c72..cd858a34bfe 100644 --- a/src/site/content/en/blog/responsiveness/index.md +++ b/src/site/content/en/blog/responsiveness/index.md @@ -167,32 +167,32 @@ over-budget amounts would be as follows: - - + + - - + + - - + + - - + + - - + +
InteractionLatencyLatency over budgetLatencyLatency over budget
Click 120 ms20 ms120 ms20 ms
Click 20 ms0 ms20 ms0 ms
Key press 60 ms10 ms60 ms10 ms
Key press 80 ms30 ms80 ms30 ms
From 10e0f6ab981c1938174efed7f23f18c9e655f7cb Mon Sep 17 00:00:00 2001 From: Philip Walton Date: Tue, 2 Nov 2021 09:46:12 -0700 Subject: [PATCH 09/11] Add code comments --- src/site/content/en/blog/responsiveness/index.md | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/src/site/content/en/blog/responsiveness/index.md b/src/site/content/en/blog/responsiveness/index.md index cd858a34bfe..d4560b254a2 100644 --- a/src/site/content/en/blog/responsiveness/index.md +++ b/src/site/content/en/blog/responsiveness/index.md @@ -142,19 +142,19 @@ using on the following thresholds for each event type: - + - + - + - +
Interaction typeBudget thresholdBudget threshold
Click/tap100 ms100 ms
Drag100 ms100 ms
Keyboard50 ms50 ms
@@ -280,6 +280,8 @@ new PerformanceObserver((entries) => { }); } } +// Set the `durationThreshold` to 50 to capture keyboard interactions +// that are over-budget (the default `durationThreshold` is 100). }).observe({type: 'event', buffered: true, durationThreshold: 50}); ``` From c230a006b04207f916cae11aeda909f759cca50e Mon Sep 17 00:00:00 2001 From: Philip Walton Date: Tue, 2 Nov 2021 11:26:42 -0700 Subject: [PATCH 10/11] Update bug hotlist link --- src/site/content/en/blog/responsiveness/index.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/src/site/content/en/blog/responsiveness/index.md b/src/site/content/en/blog/responsiveness/index.md index d4560b254a2..2c75401068d 100644 --- a/src/site/content/en/blog/responsiveness/index.md +++ b/src/site/content/en/blog/responsiveness/index.md @@ -77,8 +77,8 @@ There are pros and cons of each approach, and we'd like to collect more data and {% Aside %} For keyboard interactions, we usually measure the `keydown` and `keyup`. But - for IME, such as input methods for Chinese and Japanese, we measure the input - events between a `compositionstart` and a `compositionend`. + for IME, such as input methods for Chinese and Japanese, we measure the + `input` events between a `compositionstart` and a `compositionend`. {% endAside %} ## Aggregate all interactions per page @@ -287,7 +287,7 @@ new PerformanceObserver((entries) => { {% Aside 'caution' %} There are currently [a few - bugs](https://bugs.chromium.org/p/chromium/issues/list?q=label:responsiveness-bugs) + bugs](https://bugs.chromium.org/p/chromium/issues/list?q=label:proj-responsiveness-bugs) in Chrome that affect accuracy of the reported interaction timestamps. We are working to fix these bugs as soon as possible, and we recommend developers test these strategies in Chrome Canary to get the most accurate results. From b5cbd84a5d4829588c0b14dc8cc6aea6b6fbb375 Mon Sep 17 00:00:00 2001 From: Philip Walton Date: Tue, 2 Nov 2021 15:42:15 -0700 Subject: [PATCH 11/11] Update the high quantile approximation option 1 --- src/site/content/en/blog/responsiveness/index.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/src/site/content/en/blog/responsiveness/index.md b/src/site/content/en/blog/responsiveness/index.md index 2c75401068d..f5cd6c62555 100644 --- a/src/site/content/en/blog/responsiveness/index.md +++ b/src/site/content/en/blog/responsiveness/index.md @@ -221,8 +221,10 @@ web pages that have a lot of interactions and may be more likely to have large outliers. We've identified two potential high-quantile approximation strategies we like: -- **Option 1:** Keep track of the largest interaction over budget. After every - 50 interactions, discard the largest and replace it with the next largest. +- **Option 1:** Keep track of the largest and second-largest interactions over + budget. After every 50 new interactions, drop the largest interaction from the + previous set of 50 and add the largest interaction from the current set of 50. + The final value will be largest remaining interaction over budget. - **Option 2:** Compute the largest 10 interactions over budget and choose a value from that list depending on the total number of interactions. Given N total interactions, select the (N / 50 + 1)th largest value, or the 10th value