Align load-fast-enough-for-pwa with Core Web Vitals?

[kaycebasques]

Summary

"Fast enough" is determined by TTI: https://github.com/GoogleChrome/lighthouse/blob/master/lighthouse-core/audits/load-fast-enough-for-pwa.js#L22

Seems like this should be based off of CWV thresholds?

[philipwalton]

Looks like the current "fast enough" is not based on the "good" threshold for TTI but rather on some lower value.

In a perfect world I think we'd like to recommend "good" on all Core Web Vitals metrics, but that may be too restrictive for any PWA definition. Maybe we can say "good enough" means not getting "poor" on any Core Web Vitals metric?

[brendankenny]

That limit is super old (#1840), so it probably is time to take another look (but an update would probably have to wait for a major version change). It's kind of unfortunate we didn't coordinate it with the checklist update when it moved to web.dev, though.

Looks like the current "fast enough" is not based on the "good" threshold for TTI but rather on some lower value. In a perfect world I think we'd like to recommend "good" on all Core Web Vitals metrics, but that may be too restrictive for any PWA definition.

Right, the idea was to decouple the PWA definition from our speed recommendations. i.e. you don't have to be RAIL/CWV/whatever FAST™ to count as a PWA (the perf category is for FAST™). So the emphasis is on the "enough" part of "fast enough" :)

[brendankenny]

For 6.0 we also discussed adding metrics other than TTI in this audit, but didn't come up with any concrete ideas.

@cheneytsai was helping look at getting TBT involved here, but I don't think we had any GH issue for tracking that.

[paulirish]

We could also align with the TWA criteria of perf score 80: https://blog.chromium.org/2019/02/introducing-trusted-web-activity-for.html

[paulirish]

All possible options

1. Leave it on TTI
   • Not preferable. Doesn't align with both TWA guidelines OR CWV thresholds OR LH weighting
2. Drop hard perf criteria entirely
   • PWA could focus only on PWA/installability concerns w/ no mention of perf.
   • Chrome fast badging is a separate incentive program that gets at the same original goal..
3. Move from metrics to non-variable numbers (count of renderblocking bytes, etc)
4. Use some combo of LCP/CLS/TBT with thresholds
   • eg TBT. Aligns with LH, but threshold is under debate as it doesn't align well with observed FID at p75
5. Use perf score threshold
   • eh LH Perf > 80 will align with TWA guidance but that guidance is also under scrutiny/debate

Let's raise this with the larger PWA group.

[cheneytsai]

Couple of thoughts but agree to raise with larger group:

1. Leave it on TTI - (Not preferable) - It's not the highest weighted metric in the Speed Score, so it seems unaligned
2. Drop hard perf criteria - This seems cleaner and simpler to me (if you keep reliability around offline out). It keeps PWA about capabilities. Any system that wants to gate on LH score can just have 2 requirements: Speed > XX and has PWA badge instead of double counting speed.
3. Move to non-variable numbers - Unclear if we have a good suggestion here, and it's one step removed from reflecting user experience
   4/5. Combo with CWV or Perf Score threshold - Makes sense to me for better alignment though I think what thresholds are important. 80/100 LH would be a bit high for even being considered a PWA

[demianrenzulli]

Just wanted to add to Cheney's thoughts, that after working with several partners on PWAs, the bullet that they keep failing all the time is the performance one (TTI).

I think the problem with keeping that metric is that: (1) it has more variance than others, which can make the test show as pass/failed alternatively even during the same day (2) it's harder to improve than others, which makes it less actionable.

As a result companies end up staying in a non-PWA state endlessly (specially those which business models that depend heavily on things like ads / 3rd party scripts).

[paulirish]

We're steering towards removing the perf audit from the PWA category. Next action was on @b1tr0t.

On the Lighthouse side we'll have to figure out what we'll do with the "fast and reliable" group.