☔️LCP Interesting Cases

[patrickhulce]

Since LCP is a new metric in 6.0 I thought we might want to get ahead of some of the more challenging cases that will end up dramatically affecting scores. Inspired by the results of @connorjclark's great analysis on score movement for 11ty pages.

Very large image problem

Lighthouse accuracy: Simulated is right and observed is wrong
Field accuracy: probably mostly wrong
Example URL: https://peterthaleikis.com/
The Problem: LCP is gated on the loading a very, very large image. Observed methods will never wait long enough to see the LCP event generated by the loading of this image and will end up picking the second largest paint as the LCP even instead. Simulated method doesn't care about waiting and just correctly says that loading that image will take 3 minutes, so LCP is 3 minutes out. This is extra problematic in field use cases. Because input stops LCP computation most RUM metrics will likely pick the second largest paint as LCP as well, so all data will point to Lighthouse being wrong when it's really the only thing correctly identifying the problem.

(renderTime is ambiguously defined in the LCP spec but in the closely related element timing spec it's referred to as the equivalent of the next paint after the onload event would have fired)

Many images before LCP

Lighthouse accuracy: Observed is right and simulated is wrong
Field accuracy: probably all correct
Example URL: https://templates.netlify.com/
The Problem: Lantern-specific issue that assumes all the images before LCP were necessary when only the largest onscreen one was, this is probably worth addressing before 6.0 final as it takes otherwise very performant sites and brings them down to an LCP of 0

Observed:

Simulated:

Lantern Trace:

WAI

Lots of cases in that doc are great signs of Lighthouse catching bad perf that we were passing before, so just wanted to recognize the good cases too :)

https://2018.nejsconf.com/ for example has a very large background image that pushes out LCP even though all other metrics are fast.

[addyosmani]

cc @tdresser fyi

[brendankenny]

Example URL: https://peterthaleikis.com/
The Problem: LCP is gated on the loading a very, very large image. Observed methods will never wait long enough to see the LCP event generated by the loading of this image and will end up picking the second largest paint as the LCP even instead.

One unfortunate thing is NetworkRequest._recomputeTimesWithResourceTiming fills in endTime, so there's not much indication in our network-requests audit how often this is currently happening. e.g. for the big image in this example when using devtools throttling, I see

{
  "url": ".../static/images/generic/code1-min.jpg",
  "startTime": 1611.6690000053495,
  "endTime": 2180.4920000140555,
  "transferSize": 4968683,
  "resourceSize": 983040,
  "statusCode": 200,
  "mimeType": "image/jpeg",
  "resourceType": "Image"
}

the only real clue is the crazy big transferSize compared to the resourceSize.

@patrickhulce maybe for the TODO list, also add adding some variation of NetworkRequest.finished to the network-records audit details?

[patrickhulce]

maybe for the TODO list, also add adding some variation of NetworkRequest.finished to the network-records audit details?

SGTM 👍

[patrickhulce]

Lighthouse Team Action Items:

• Add a toplevel warning when we hit the loading timeout
• Fail with LATE_LCP in the observed case where we hit our loading timeout and images didn't finish loading
• Filter out more images from LCP in lantern (unclear how, open a draft PR to decide after more research. initial ideas: rely on ImageElements or find a way to link up the trace event) core(lantern): improve LCP tail error #10529
• Expose NetworkRequest.finished in our network-requests audit core: include finished state on hidden network-requests audit #10530

[patrickhulce]

@tdresser there's a lot of noise here, but the bit of this issue that's relevant to you is the "Very large image" case where we're concerned about field data.

Field accuracy: probably mostly wrong
Example URL: https://peterthaleikis.com/
The Problem: LCP is gated on the loading of a very, very large image. Observed methods will never wait long enough to see the LCP event generated by the loading of this image and will end up picking the second largest paint as the LCP event instead. This is extra problematic in field use cases. Because input stops LCP computation most RUM metrics will likely pick the second largest paint as LCP as well, so all data will point to Lighthouse being wrong when it's really the only thing correctly identifying the problem.

We checked at the URL that gets a legit 170s LCP in the lab on a phone is seeing >60% LCP meeting our thresholds in the field, presumably due to this issue.

[tdresser]

@anniesullie @npm1

[patrickhulce]

One other note on LCP we were thinking about for Chrome Speed Metrics folks, the progressive loading of images doesn't really jive well with the experience reported by LCP. In the 170s LCP case, that's when the image is "done", but because it's a progressive JPEG (with a massive resolution well beyond what's needed for its displayed size) it looks loaded from the user's perspective much, much earlier because the earlier passes are sufficient for the displayed resolution.

It'd probably greatly complicate the definition to try to handle this case, but just another data point that maybe the load event of the image is perhaps too late in some cases.

[tdresser]

I think in that case, the metric still incentivizes the right behavior, so I'm not toooo concerned about it.
Does that seem reasonable to you?

[patrickhulce]

the metric still incentivizes the right behavior, so I'm not toooo concerned about it.

That was roughly our thinking too, so we're not overly worried about it either. Just anticipating the inevitable bug reports when folks come to us saying 170s is ridiculous and we mostly agree :) haha

That being said, if earlier triggers of the bad LCP happened because of progressive loading consideration (i.e. the bad LCP case fires at 15s instead of 170s), it might reduce the severity of the false reporting problem because more users are likely to hit the LCP performance entry in the field. I don't think this is necessary the right way to solve that problem, but something to think about.

FWIW, I don't think changes to the underlying metric itself are necessary to address the field reporting problem. I think the solution lies somewhere in the reporting space with an absolute cutoff ala TTI and a "DID_NOT_FINISH" type result for LCP or something similar.

[npm1]

One possibility to improve this problem for RUM data gathering is for the LCP API to fire some signal that indicates that the current largest is an image that has not yet finished loading. Right now there is no easy way to know.

[patrickhulce]

That signal would definitely help. I'm still a little concerned about folks learning to implement this correctly. You mentioned in w3c/largest-contentful-paint#50 (comment) that this shouldn't be a problem in CrUX. Is there an end signal that's being used we could adapt and make available to PerformanceObservers and/or Lighthouse that might simplify things?

[npm1]

Perhaps the first question to resolve here is: what is the correct reporting behavior in the case of a largest image that did not finish loading? We think this case should be reported as 'unknown LCP'. WDYT?

[patrickhulce]

We think this case should be reported as 'unknown LCP'.

Agreed. "unknown", "did not finish", anything of the sort that doesn't lead someone to incorrectly count the first paint and think LCP was 2s.

[paulirish]

Lighthouse has no remaining AIs here. CSM will carry the torch on handling these.