New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Lighthouse CLI on Circle CI scores higher than expected #4003

Closed
ataylorme opened this Issue Dec 6, 2017 · 4 comments

Comments

Projects
None yet
3 participants
@ataylorme

ataylorme commented Dec 6, 2017

Using the Chome extension, in incognito mode with other extensions disabled, I am getting scores in the mid-80's.

I added Lighthouse to my continuous integration process tests running on the live website and a new environment created for pull requests. The idea being if the overall performance score is more than 5 points worse on the pull request the Circle CI build will fail, disallowing a merge on GitHub.

To test this is intentionally created a pull request that should receive a lower score by adding a large image to the page and disable Varnish full page cache so the response is served directly from WordPress.

As expected, the Lighthouse report run from my local machine using the Chrome extension returns a performance score of 86 with a first paint of3,120ms.

screen shot 2017-12-06 at 3 35 40 pm

Unexpectedly, the Lighthouse report run on Circle CI using the CLI version returns a performance score of 100 with a first paint of770ms.

screen shot 2017-12-06 at 3 43 18 pm

The Circle CI log for the job running the Lighthouse test can be found here and the main repository code is open source and available here.

I expected something running on a CI server to be slightly faster but the large discrepancy (2,350ms) in first paint time and the resulting perfect performance score, for a URL that has known issues, is causing my desired use case of checking the score as part of a CI process to be ineffective.

Any suggestions are most welcome.

@patrickhulce

This comment has been minimized.

Collaborator

patrickhulce commented Dec 7, 2017

Thanks for reporting @ataylorme great to see you're adding Lighthouse to your CI! 🎉 👏

It looks like your code is running Lighthouse on headless where throttling is not available. For this reason, we actually recommend running regular headful Chrome with XVFB in server environments (see the docs).

Let us know if you run into any issues getting XVFB running or you find something in those docs to be unclear!

related #3997 #2008 #2675

@ataylorme

This comment has been minimized.

ataylorme commented Dec 7, 2017

Thanks, @patrickhulce for the quick reply! The scores make more sense with no throttling in headless mode. Is there a maintained Docker image with Chrome, Lighthouse and XVFB I can use?

@ataylorme ataylorme closed this Dec 7, 2017

@ebidel

This comment has been minimized.

Member

ebidel commented Dec 8, 2017

You can use https://github.com/ebidel/lighthouse-ci. Working on bringing this work in officially to this repo: #3715

@ataylorme

This comment has been minimized.

ataylorme commented Dec 8, 2017

Great, thanks @ebidel. I've subscribed to that thread. I was able to use https://github.com/ebidel/lighthouse-ci in the meantime. Thank you for the work there! Really comprehensive.

Once a Docker image comes out I'll switch over to running Lighthouse locally in Circle CI.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment