Chrome crashes when running stacks gatherer sometimes

[patrickhulce]

Filed from GoogleChrome/lighthouse-ci#381 (comment)

Provide the steps to reproduce

1. Lots of complicated infrastructure setup described in Feature: Specific exit codes depending on failure lighthouse-ci#381 (comment) :(

AWS Lightsail Node-js server(v12.6.1) at 4 GB RAM, 2 vCPUs, 80 GB SSD.
The folllowing exact commands were run:
1 mkdir node-apps
2 cd node-apps
3 touch apps.js
4 edit apps.js to initiate a server on port 3000
const http = require('http') ;
const server = http.createServer((req,res) => {
res.end("Server Running!") ;
})cd
server.listen(3000) ;
5 npm init - complete the wizard
6 sudo npm install -g lighthouse
6 sudo npm install -g chromium-launcher
7 sudo apt install chromium-browser
8 chromium-browser --product-version
9 sudo npm install -g tcp-port-used
the following lighthouse cli was run
lighthouse https://www.etihad.com/ --chrome-flags="--headless" --output json --output ht
ml --output-path ./myfile.json --collect.settings.disableStorageReset --verbose

What is the current behavior? Chrome appears to crash and then Lighthouse exits with the dreaded protocol took too long error.

What is the expected behavior? Lighthouse returns results.

Environment Information

• Affected Channels: CLI
• Lighthouse version: v6.1.1
• Chrome version: ?
• Node.js version: ?
• Operating System: ?

Related issues
ref #6512

I have been unable to reproduce this error so far, but Chrome crashing in the middle probably should result in a more specific error message with Chrome log output rather than the dreaded protocol message.

[dimension85]

As requested:
lsb_release -a
No LSB modules are available.
Distributor ID:Ubuntu
Description:Ubuntu 16.04.6 LTS
Release:16.04
Codename:xenial

$ chromium-browser --product-version
81.0.4044.138

Lighthouse runs shows following environment as
"userAgent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/83.0.4103.61 Safari/537.36",
"environment": {
"networkUserAgent": "Mozilla/5.0 (Linux; Android 7.0; Moto G (4)) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4143.7 Mobile Safari/537.36 Chrome-Lighthouse",
"hostUserAgent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/83.0.4103.61 Safari/537.36",
},
"lighthouseVersion": "6.1.1",

[patrickhulce]

Hm, something is strange here @dimension85 .

Lighthouse was using 83.0.4103.61 not 81.0.4044.138 according to that environment information. Are these all from the same machine? How were you able to get the environment information if you always run into the PROTOCOL__TIMEOUT error?

[dimension85]

They are from the same environment and consistent across all the environments.
The instructions I provided on the other thread deliver the environment as it is. This is recreate-able by anyone. Everything is native as delivered If there is something I can try to change it and rerun then I can try but I am running CLI now to help with this and get the same results.
I am able to get the runtime UA strings from successful runs on other URLs that work and as they are consistent I have assumed that they are the same for this test as well

[patrickhulce]

OK thanks @dimension85

I tried with Lightsail running exactly these steps and I can reproduce a PROTOCOL__TIMEOUT now 👍

> sudo npm install -g lighthouse
...
> sudo apt-get install chromium-browser
...
>  chromium-browser --product-version
81.0.4044.138
> lighthouse https://example.com --chrome-flags=--headless --output=json --output-path=test.json --verbose
... output works but fails some gatherers with unknown protocol Audits.enable
> lighthouse https://www.etihad.com/  --chrome-flags=--headless --output=json  --output-path=test.json --verbose --only-audits=interact
ive
fails with the PROTOCOL__TIMEOUT on stacks gatherer every time (60s timeout)

Installing google-chrome-stable did not help either :/ From what I can tell though this only affects Lightsail. I could not reproduce with regular EC2 instance or GCP or any other environment I had access to.

[patrickhulce]

Verbose logs seem to indicate it's a hanging network request for https://www.etihad.com/asset-manifest.json issued by the stacks gatherer

  method => browser:verbose Runtime.evaluate {"expression":"(function wrapInNativePromise() {\n        const __nativePromise = window.__nativePromise || Promi +3ms
  <= event:verbose Debugger.scriptParsed {"scriptId":"203","url":"","startLine":0,"startColumn":0,"endLine":1804,"endColumn":10,"executionContextId":3,"hash": +34ms
  <= event:verbose Network.requestWillBeSentExtraInfo {"requestId":"31694.235","associatedCookies":[{"blockedReasons":[],"cookie":{"name":"akacd_PR1","value": +7ms
  <= event:verbose Network.requestWillBeSent {"requestId":"31694.235","loaderId":"C668DA50DF88FCA1E9AD605357CDBADA","documentURL":"https://www.etihad.com/en-u +2ms
>>>>> PROTOCOL__TIMEOUT ERROR  <<<<<<

[dimension85]

sounds like your making progress - is there a workaround or specific audit related to this that I can suppress for now?

[patrickhulce]

specific audit related to this that I can suppress for now?

Unfortunately, no. The stacks gatherer is run on every run single invocation of Lighthouse without a way to skip it :/

is there a workaround

The workaround from my perspective is run Lighthouse in literally any other environment than Lightsail 😆 I couldn't reproduce it anywhere but there.

[dimension85]

Hmm, I think your logic is the wrong way around as you are suggesting I move from a stable environment that has a repeatable and fixable error on it to one that has a random and unable to diagnose error on it? Perhaps everyone should move to Lightsail so the problem could be diagnosed? The fact you cannot replicate it does not suggest it does not happen as thread 6512 shows /lighthouse-core/gather/driver.js:352:21 has occurred for others as well - are these the same or similar issues?

[patrickhulce]

to one that has a random and unable to diagnose error on it

The incidence rate in other CLI environments is very low. There are many, many services that run Lighthouse in other cloud environments that don't run into this. In our own infrastructure in GCP this error occurs <1% of the time where a retry is more than reasonable. Generally, yes, I would consider moving from an environment that fails 100% of the time to one that fails <1% to be a decent temporary workaround.

The fact you cannot replicate it does not suggest it does not happen as thread 6512 shows /lighthouse-core/gather/driver.js:352:21

That is the location for the umbrella error of Chrome took too long to respond. We are well aware that this error occurs. The problem is that it can occur for 1000 different reasons. We are grateful you have been the first person to ever find a specific URL in a specific environment where it always occurs due to a very specific reason and we will take steps to fix that specific source. Most of the time though this happens for sporadic, transient, and unrelated-to-Lighthouse reasons that aren't going away and PROTOCOL__TIMEOUT will continue to happen.

[dimension85]

I have also seen other random PROTOCOL__TIMEOUT events which I have found a retry resolves, so was surprised when this occurred.

I appreciate the effort and priority you have applied to this and look forward to the fix when it becomes available.

[patrickhulce]

Huzzah, we've found an even more reproducible example of this exact same issue, but it occurs in every environment! 🎉

https://www.tiffany.com/ gets stuck in the stacks gatherer in PROTOCOL_TIMEOUT every time, in every environment I tested. This will make a fix much easier to test 👍