WPT does not apply CPU throttling to Lighthouse run

[pmd-bgermain]

Summary

I tried some tests on multiple cases with Speedcurve and Webpagetest and found some troubling behaviors :

I used the configuration for CPU throttling and set it to 4, 6 and 12 on the same page with same 3G connection configuration and got performance score of respectively : 38, 43 and 49.

I expected a completely different results with a lower score as the throttling get higher.

I found that when using the throttling I have errors on javascript and CPU related diagnostics:

• JavaScript execution time Error!
• Minimizes main-thread work Error!
• Third-Party Usage Error!
  It may or may not be related but what I thought when I saw this is that the when CPU is slow, execution of javascript and main thread go in timeout mode .... and can't achieve any measure.

It could be nice to have more information on that when using the tool so user can understand what happens.

Best,

[jcrm70]

I don't know if it will help you, but I asked myself about this a few weeks ago, and they answered me here:

#9544 (comment)

I think that although the start in your case is different, the problem is the same.

and this the origen of the problem:

#7764

[patrickhulce]

It may or may not be related but what I thought when I saw this is that the when CPU is slow, execution of javascript and main thread go in timeout mode .... and can't achieve any measure.

This is expected. When you increase the throttling such that the page never reaches TTI, we'll timeout trying to wait for it to happen.

I used the configuration for CPU throttling and set it to 4, 6 and 12 on the same page with same 3G connection configuration and got performance score of respectively : 38, 43 and 49.

Can you share the page and/or test URLs? The score should get indeed get lower and/or timeout as the throttling increases.

I found that when using the throttling I have errors on javascript and CPU related diagnostics:

That is likely #7764 as @jcrm70 notes and is (mostly) unrelated to the use of throttling.

[pmd-bgermain]

Thanks for the feedback @jcrm70, you're right the problem is the same as yours even if my ticket seems to list two different issues now you're pointing that ...

@patrickhulce Sure I can share my lighthouse result pages, here they are :

• Throttling of 4 > https://wpt.speedcurve.com/lighthouse.php?test=190910_D8_ae09447bda097fc02464a1fe0a29e697&run=3
• Throttling of 6 > https://wpt.speedcurve.com/lighthouse.php?test=190911_3B_5aa4612061b9177c6b559fa8eaa0e370&run=2
• Throttling of 12 > https://wpt.speedcurve.com/lighthouse.php?test=190911_2P_ee53a45af7f2c89709f4f2b2a089916a&run=3

I tried them on a private session without logging and seems accessible with the link only. Don't hesitate to tell me if you have trouble accessing them, I will do the test on WPT.

[patrickhulce]

Thanks @pmd-bgermain! Ah those look like the results of stock WPT integration with Lighthouse which, AFAIK, always uses Lighthouse recommended throttling settings regardless of what is configured in WPT metrics. Notice that the "CPU/Memory Power" metric in runtime settings actually increases significantly between 4x and 12x which is an indication that the view LH has is of a more powerful CPU in the 12x throttled run.

The differences you're seeing here then is likely just a result of normal variability. Thanks for filing #9700 btw :)

[brendankenny]

Looks like this issue was addressed. If not, please comment and we can reopen.

[zeman]

Hi @patrickhulce, those WPT results were on SpeedCurve where I believe both network and CPU throttling are disabled in Lighthouse. If you look at the Lighthouse log the following options were set --disable-network-throttling --disable-cpu-throttling --throttling-method provided and at the bottom of the Lighthouse report is say the CPU and network throttling were "Provided by environment". My understanding is that this means Lighthouse has all throttling turned off and instead the "environment" which in this case is WebPageTest is then throttling the network and CPU at the OS layer. Is that the wrong interpretation?

[patrickhulce]

My understanding is that this means Lighthouse has all throttling turned off and instead the "environment" which in this case is WebPageTest is then throttling the network and CPU at the OS layer. Is that the wrong interpretation?

tl;dr - yes and no. AFAICT the OS-level CPU throttling is not being applied to the Lighthouse run in certain circumstances.

The throttling for Lighthouse integration with WPT is indeed always controlled by WPT not Lighthouse. However, the code that I linked to in WPT is referencing how WPT always uses the "Fast 3G" traffic shaping settings for the Lighthouse run regardless of what was configured for the rest of the WPT run (it looks like it's been modified more recently though to be skipped if lighthouseThrottle is set for the job, so maybe you're already using that feature?)

Regardless of the traffic shaping though, it appears that CPU throttling settings from the device in WPT do not always get set for the Lighthouse run.

Doing a quick test run on an emulated Moto E, the WPT metrics show successful CPU throttling (FCP ~8s, TTI ~120s) while the Lighthouse report does not seem to inherit any of that CPU throttling (FCP ~3s, TTI ~40s).

Additionally, if you compare the trace of the two runs (WPT LH), the task durations are off by about a multiple of ~6x or more. (note the ~225ms layout of 2034 objects vs. ~1.9s layout of 2034 objects and 760ms timer fire vs. 4.4s timer fire)

This would backup the results from this issue that the observed CPU power from Lighthouse has no connection to the WPT CPU throttling level.

It seems like we might want to change the Lighthouse command in WPT to something along the lines of

'lighthouse --throttling-method=devtools --throttling.requestLatencyMs=0 --throttling.downloadThroughputKbps=0 --throttling.uploadThroughputKbps=0 --throttling.cpuSlowdownMultiplier=%s' % self.job['throttle_cpu'] if not self.options.android else 1 and/or fix the OS-level application of CPU throttling for the Lighthouse run?

cc @pmeenan if this sounds accurate or if I'm missing something.

[pmeenan]

Looks like the WebPageTest path when emulating mobile devices was not applying the CPU throttling. Just pushed a fix that should be rolling out over the next hour.

In WPT, the throttling is always applied externally to lighthouse (usually at the OS layer). The CPU throttling in particular is calibrated to be more consistent across different machines so it is not a stock throttle value but it should match the device that is being emulated and should be consistent with the WPT test (once the fix is rolled out).

[patrickhulce]

woohoo thanks a million pat! ❤️

[paulirish]

I did a followup test to @patrickhulce and not sure what to make of the numbers...

WPT test for the verge.

• trace of WPT run
• trace of WPT LH run

The task lengths in WPT are ~2.5x bigger than the numbers for Lighthouse. This is much better than the 6x that patrick saw before, but it still doesn't make sense to me.

Screenshot showing the evaluatescript for concert_ads.js in both traces . (353ms vs 128ms):

@pmeenan do you know what's going on here? is it the difference between cgroup vs devtools CPU throttling?

Also reading your september fix and following the logic in wptagent, I couldn't figure out why this doesn't lead to double-throttling in the Lighthouse + --throttling case. Could you help me understand why not? :) (Obviously, the runs above indicate the WPT run is definitely slower so it makes sense that LH isn't getting double throttled in reality.)

---

Small edit. I reran theverge on a real motog4 (no emulation, obv) and the task lengths are roughly the same. Mostly seeing LH tasks as 10% larger. Seems okay.