Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WPT does not apply CPU throttling to Lighthouse run #9668

Closed
pmd-bgermain opened this issue Sep 13, 2019 · 9 comments
Closed

WPT does not apply CPU throttling to Lighthouse run #9668

pmd-bgermain opened this issue Sep 13, 2019 · 9 comments

Comments

@pmd-bgermain
Copy link

@pmd-bgermain pmd-bgermain commented Sep 13, 2019

Summary

I tried some tests on multiple cases with Speedcurve and Webpagetest and found some troubling behaviors :

I used the configuration for CPU throttling and set it to 4, 6 and 12 on the same page with same 3G connection configuration and got performance score of respectively : 38, 43 and 49.

I expected a completely different results with a lower score as the throttling get higher.

I found that when using the throttling I have errors on javascript and CPU related diagnostics:

  • JavaScript execution time Error!
  • Minimizes main-thread work Error!
  • Third-Party Usage Error!
    It may or may not be related but what I thought when I saw this is that the when CPU is slow, execution of javascript and main thread go in timeout mode .... and can't achieve any measure.

It could be nice to have more information on that when using the tool so user can understand what happens.

Best,

@jcrm70

This comment has been minimized.

Copy link

@jcrm70 jcrm70 commented Sep 13, 2019

I don't know if it will help you, but I asked myself about this a few weeks ago, and they answered me here:

#9544 (comment)

I think that although the start in your case is different, the problem is the same.

and this the origen of the problem:

#7764

@patrickhulce

This comment has been minimized.

Copy link
Collaborator

@patrickhulce patrickhulce commented Sep 13, 2019

It may or may not be related but what I thought when I saw this is that the when CPU is slow, execution of javascript and main thread go in timeout mode .... and can't achieve any measure.

This is expected. When you increase the throttling such that the page never reaches TTI, we'll timeout trying to wait for it to happen.

I used the configuration for CPU throttling and set it to 4, 6 and 12 on the same page with same 3G connection configuration and got performance score of respectively : 38, 43 and 49.

Can you share the page and/or test URLs? The score should get indeed get lower and/or timeout as the throttling increases.

I found that when using the throttling I have errors on javascript and CPU related diagnostics:

That is likely #7764 as @jcrm70 notes and is (mostly) unrelated to the use of throttling.

@pmd-bgermain

This comment has been minimized.

Copy link
Author

@pmd-bgermain pmd-bgermain commented Sep 16, 2019

Thanks for the feedback @jcrm70, you're right the problem is the same as yours even if my ticket seems to list two different issues now you're pointing that ...

@patrickhulce Sure I can share my lighthouse result pages, here they are :

I tried them on a private session without logging and seems accessible with the link only. Don't hesitate to tell me if you have trouble accessing them, I will do the test on WPT.

@patrickhulce

This comment has been minimized.

Copy link
Collaborator

@patrickhulce patrickhulce commented Sep 23, 2019

Thanks @pmd-bgermain! Ah those look like the results of stock WPT integration with Lighthouse which, AFAIK, always uses Lighthouse recommended throttling settings regardless of what is configured in WPT metrics. Notice that the "CPU/Memory Power" metric in runtime settings actually increases significantly between 4x and 12x which is an indication that the view LH has is of a more powerful CPU in the 12x throttled run.

The differences you're seeing here then is likely just a result of normal variability. Thanks for filing #9700 btw :)

@brendankenny

This comment has been minimized.

Copy link
Member

@brendankenny brendankenny commented Sep 24, 2019

Looks like this issue was addressed. If not, please comment and we can reopen.

@zeman

This comment has been minimized.

Copy link
Contributor

@zeman zeman commented Sep 25, 2019

Hi @patrickhulce, those WPT results were on SpeedCurve where I believe both network and CPU throttling are disabled in Lighthouse. If you look at the Lighthouse log the following options were set --disable-network-throttling --disable-cpu-throttling --throttling-method provided and at the bottom of the Lighthouse report is say the CPU and network throttling were "Provided by environment". My understanding is that this means Lighthouse has all throttling turned off and instead the "environment" which in this case is WebPageTest is then throttling the network and CPU at the OS layer. Is that the wrong interpretation?

@patrickhulce

This comment has been minimized.

Copy link
Collaborator

@patrickhulce patrickhulce commented Sep 25, 2019

My understanding is that this means Lighthouse has all throttling turned off and instead the "environment" which in this case is WebPageTest is then throttling the network and CPU at the OS layer. Is that the wrong interpretation?

tl;dr - yes and no. AFAICT the OS-level CPU throttling is not being applied to the Lighthouse run in certain circumstances.

The throttling for Lighthouse integration with WPT is indeed always controlled by WPT not Lighthouse. However, the code that I linked to in WPT is referencing how WPT always uses the "Fast 3G" traffic shaping settings for the Lighthouse run regardless of what was configured for the rest of the WPT run (it looks like it's been modified more recently though to be skipped if lighthouseThrottle is set for the job, so maybe you're already using that feature?)

Regardless of the traffic shaping though, it appears that CPU throttling settings from the device in WPT do not always get set for the Lighthouse run.

Doing a quick test run on an emulated Moto E, the WPT metrics show successful CPU throttling (FCP ~8s, TTI ~120s) while the Lighthouse report does not seem to inherit any of that CPU throttling (FCP ~3s, TTI ~40s).

Additionally, if you compare the trace of the two runs (WPT LH), the task durations are off by about a multiple of ~6x or more. (note the ~225ms layout of 2034 objects vs. ~1.9s layout of 2034 objects and 760ms timer fire vs. 4.4s timer fire)

This would backup the results from this issue that the observed CPU power from Lighthouse has no connection to the WPT CPU throttling level.

It seems like we might want to change the Lighthouse command in WPT to something along the lines of

'lighthouse --throttling-method=devtools --throttling.requestLatencyMs=0 --throttling.downloadThroughputKbps=0 --throttling.uploadThroughputKbps=0 --throttling.cpuSlowdownMultiplier=%s' % self.job['throttle_cpu'] if not self.options.android else 1 and/or fix the OS-level application of CPU throttling for the Lighthouse run?

cc @pmeenan if this sounds accurate or if I'm missing something.

@patrickhulce patrickhulce reopened this Sep 25, 2019
@patrickhulce patrickhulce changed the title Consistenly interactive / NO_TTI_NETWORK_IDLE_PERIOD WPT does not apply CPU throttling to Lighthouse run Sep 25, 2019
@pmeenan

This comment has been minimized.

Copy link

@pmeenan pmeenan commented Sep 25, 2019

Looks like the WebPageTest path when emulating mobile devices was not applying the CPU throttling. Just pushed a fix that should be rolling out over the next hour.

In WPT, the throttling is always applied externally to lighthouse (usually at the OS layer). The CPU throttling in particular is calibrated to be more consistent across different machines so it is not a stock throttle value but it should match the device that is being emulated and should be consistent with the WPT test (once the fix is rolled out).

@patrickhulce

This comment has been minimized.

Copy link
Collaborator

@patrickhulce patrickhulce commented Sep 25, 2019

woohoo thanks a million pat! ❤️

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
6 participants
You can’t perform that action at this time.