Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ci(docs-infra): `test_aio`/`test_aio_local` jobs occasionally fail due to low PWA score #29910

Closed
gkalpak opened this Issue Apr 15, 2019 · 5 comments

Comments

Projects
1 participant
@gkalpak
Copy link
Member

commented Apr 15, 2019

Recently, the test_aio/test_aio_local jobs started flaking relatively often due to the PWA score being too low (71% vs 95% required).

Example failures: 284426, 284459, 284483, 284494, 284556, 284583.

Sample output:

Audit scores:
  - Performance:          42 / 100
  - Accessibility:        94 / 100
  - Best Practices:       93 / 100
  - SEO:                  82 / 100
  - Progressive Web App:  71 / 100

Lighthouse PWA score:
  - Expected:  95 / 100 (or higher)
  - Actual:    71 / 100

Error: PWA score is too low. (71 < 95)

For reference, here are two Lighthouse reports, one with PWA score 100% and one with 71%:

@ngbot ngbot bot added this to the needsTriage milestone Apr 15, 2019

@gkalpak gkalpak added this to To do in dev-infra via automation Apr 15, 2019

gkalpak added a commit to gkalpak/angular that referenced this issue Apr 15, 2019

ci(docs-infra): temporarily lower the min required PWA score for loca…
…lhost tests

The PWA score tests have been occasionally failing on CI recently
(possibly due to CI VM/network issues).

This commit temporarily disables them, until we investigate the
root-cause and/or put a work-around in place.

The PWA score tests are still run against the deployed versions (which
don't suffer as much) or PRs with public previews (as part of the
`test_aio_preview` job) and on upstream builds (as part of the
`deploy_aio` job).

Related to angular#29910.

[1]: https://github.com/angular/angular/blob/3a836c362/.circleci/config.yml#L390

alxhub added a commit that referenced this issue Apr 15, 2019

ci(docs-infra): temporarily lower the min required PWA score for loca…
…lhost tests (#29911)

The PWA score tests have been occasionally failing on CI recently
(possibly due to CI VM/network issues).

This commit temporarily disables them, until we investigate the
root-cause and/or put a work-around in place.

The PWA score tests are still run against the deployed versions (which
don't suffer as much) or PRs with public previews (as part of the
`test_aio_preview` job) and on upstream builds (as part of the
`deploy_aio` job).

Related to #29910.

[1]: https://github.com/angular/angular/blob/3a836c362/.circleci/config.yml#L390

PR Close #29911

matsko pushed a commit to matsko/angular that referenced this issue Apr 15, 2019

ci(docs-infra): temporarily lower the min required PWA score for loca…
…lhost tests (angular#29911)

The PWA score tests have been occasionally failing on CI recently
(possibly due to CI VM/network issues).

This commit temporarily disables them, until we investigate the
root-cause and/or put a work-around in place.

The PWA score tests are still run against the deployed versions (which
don't suffer as much) or PRs with public previews (as part of the
`test_aio_preview` job) and on upstream builds (as part of the
`deploy_aio` job).

Related to angular#29910.

[1]: https://github.com/angular/angular/blob/3a836c362/.circleci/config.yml#L390

PR Close angular#29911
@gkalpak

This comment has been minimized.

Copy link
Member Author

commented Apr 16, 2019

By comparing the reports, the difference is that the SW is not detected in the failing runs (possibly due to it being registered after the app stabilizes).
Related Lighthouse issues: #2924, #5527, #6342

@gkalpak

This comment has been minimized.

Copy link
Member Author

commented Apr 16, 2019

It turned out to be related to the old Chrome version used on CI (v70). Switching to puppeteer@1.14.0 (which brings in the equivalent of Chrome/75.0.3738.0), makes the tests pass reliably.

(Now we just need to figure out what is the best way to update Chrome across our CI jobs 😁)

EDIT:
Here is a PoC of running the PWA score tests against the latest Chrome version (but we should have a more generic way - potentially based on puppeteer).

EDIT (2):
Actually, that doesn't seem to be entirely true: #29921 (comment) 😞

gkalpak added a commit to gkalpak/angular that referenced this issue Apr 16, 2019

gkalpak added a commit to gkalpak/angular that referenced this issue Apr 16, 2019

gkalpak added a commit to gkalpak/angular that referenced this issue Apr 16, 2019

@gkalpak

This comment has been minimized.

Copy link
Member Author

commented Apr 16, 2019

Capturing some more info for future reference:

It seems that the failurs are somehow related to the particular job instance/VM instance. For example, ssh'ing into a failing job/VM (e.g. #29921's 285524), will consistently fail the PWA tests. (Consistently, as in 50/50 failures.)

In that case, Lighthouse fails to detect the SW (and I indeed see from the server logs that the SW script was not requested). At the same time, loading the target URL in the same puppeteer instance, I can see the SW script being loaded and executing correctly (based on the server logs). For example:

// Open a browser.
const browser = await puppeteer.launch({defaultViewport: null, headless: false});
const flags = {port: new URL(browser.wsEndpoint()).port};

// Open a page on the same browser to see if the SW is loaded.
const page = await browser.newPage();
await page.goto(url);
await new Promise(resolve => setTimeout(resolve, 10000));  // SW is loaded :)
await p.close();

// Run Lighthouse.
await lighthouse(url, flags, config);  // SW is not loaded :(

(FWIW, I even tried removing node_modules/ (both in root and in aio/), re-running yarn --cwd=aio build and nothing changed 😞)

@gkalpak

This comment has been minimized.

Copy link
Member Author

commented Apr 16, 2019

For completeness:
Ssh'ing into a passing job/VM (e.g. #29921's 285531) and running the PWA tests failed 2/15 times (i.e. still failed but with a much lower fewquency) 🤷‍♂️

gkalpak added a commit to gkalpak/angular that referenced this issue Apr 17, 2019

ci(docs-infra): increase wait for SW on localhost to avoid CI flakes
The local server is unoptimized (e.g. no compression), so wait longer
for the SW to be loaded and registered to allow Lighthouse reliably
detect it, especially on slower environments (e.g. CI).

Fixes angular#29910

gkalpak added a commit to gkalpak/angular that referenced this issue Apr 17, 2019

ci(docs-infra): increase wait for SW on localhost to avoid CI flakes
The local server is unoptimized (e.g. no compression), so wait longer
for the SW to be loaded and registered to allow Lighthouse reliably
detect it, especially on slower environments (e.g. CI).

Fixes angular#29910

gkalpak added a commit to gkalpak/angular that referenced this issue Apr 17, 2019

ci(docs-infra): increase wait for SW on localhost to avoid CI flakes
The server used for testing on localhost has less optimizations (e.g.
serves uncompressed files), so we need to wait longer the ServiceWorker
to be loaded and registered to allow Lighthouse to reliably detect it,
especially on slower environments (e.g. CI).

Fixes angular#29910

gkalpak added a commit to gkalpak/angular that referenced this issue Apr 17, 2019

ci(docs-infra): increase wait for SW on localhost to avoid CI flakes
The server used for testing on localhost has less optimizations (e.g.
serves uncompressed files), so we need to wait longer the ServiceWorker
to be loaded and registered to allow Lighthouse to reliably detect it,
especially on slower environments (e.g. CI).

Fixes angular#29910

gkalpak added a commit to gkalpak/angular that referenced this issue Apr 17, 2019

ci(docs-infra): increase wait for SW on localhost to avoid CI flakes
The server used for testing on localhost has less optimizations (e.g.
serves uncompressed files), so we need to wait longer the ServiceWorker
to be loaded and registered to allow Lighthouse to reliably detect it,
especially on slower environments (e.g. CI).

Related: GoogleChrome/lighthouse#5527 (comment)

Fixes angular#29910

gkalpak added a commit to gkalpak/angular that referenced this issue Apr 17, 2019

ci(docs-infra): increase wait for SW on localhost to avoid CI flakes
The server used for testing on localhost has less optimizations (e.g.
serves uncompressed files), so we need to wait longer the ServiceWorker
to be loaded and registered to allow Lighthouse to reliably detect it,
especially on slower environments (e.g. CI).

Related: GoogleChrome/lighthouse#5527 (comment)

Fixes angular#29910
@gkalpak

This comment has been minimized.

Copy link
Member Author

commented Apr 17, 2019

We have a work-around (thx to GoogleChrome/lighthouse#5527 (comment)) 🎉
PR: #29953

gkalpak added a commit to gkalpak/angular that referenced this issue Apr 17, 2019

ci(docs-infra): increase wait for SW on localhost to avoid CI flakes
The server used for testing on localhost has less optimizations (e.g.
serves uncompressed files), so we need to wait longer the ServiceWorker
to be loaded and registered to allow Lighthouse to reliably detect it,
especially on slower environments (e.g. CI).

Related: GoogleChrome/lighthouse#5527 (comment)

Fixes angular#29910

wKoza added a commit to wKoza/angular that referenced this issue Apr 17, 2019

ci(docs-infra): temporarily lower the min required PWA score for loca…
…lhost tests (angular#29911)

The PWA score tests have been occasionally failing on CI recently
(possibly due to CI VM/network issues).

This commit temporarily disables them, until we investigate the
root-cause and/or put a work-around in place.

The PWA score tests are still run against the deployed versions (which
don't suffer as much) or PRs with public previews (as part of the
`test_aio_preview` job) and on upstream builds (as part of the
`deploy_aio` job).

Related to angular#29910.

[1]: https://github.com/angular/angular/blob/3a836c362/.circleci/config.yml#L390

PR Close angular#29911

benlesh added a commit that referenced this issue Apr 17, 2019

ci(docs-infra): increase wait for SW on localhost to avoid CI flakes (#…
…29953)

The server used for testing on localhost has less optimizations (e.g.
serves uncompressed files), so we need to wait longer the ServiceWorker
to be loaded and registered to allow Lighthouse to reliably detect it,
especially on slower environments (e.g. CI).

Related: GoogleChrome/lighthouse#5527 (comment)

Fixes #29910

PR Close #29953

@benlesh benlesh closed this in 60e9d2d Apr 17, 2019

dev-infra automation moved this from To do to Done Apr 17, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.