Lighthouse does not work with a CSP with sandbox without allow-scripts

[Gunni]

Provide the steps to reproduce

1. Run LH on https://meh.is/

What is the current behavior?

Test will time out

Chrome console will print: Blocked script execution in 'https://meh.is/' because the document's frame is sandboxed and the 'allow-scripts' permission is not set.

What is the expected behavior?

Lighthouse should work despite website CSP since it is part of the browser.

Environment Information

• Affected Channels: DevTools, Extension
• Lighthouse version:
• Chrome version: 87.0.4280.88 (Official Build) (64-bit) (cohort: Stable)
• Node.js version:
• Operating System: Windows 10 1909

[adamraine]

So there are two PROTOCOL_TIMEOUTs I noticed:

Page.evaluate: Execution can time out when setTimeout is used in the expression (e.g. resolving a promise after a timeout) because the body of the setTimeout call is blocked by any CSP with sandbox;. This issue is not specific to CDP, using setTimeout directly in the console on https://meh.is will show a "Blocked script execution..." error. My theory is that expressions run in the console have special privileges to bypass the CSP, but any expression added to the event queue will not have the same privileges. I came up with a few possible solutions:

• Have DT allow the body of setTimeout to inherit the CSP privileges of the caller. We will need to find out if the current behavior is intentional.
• Avoid using setTimeout in our expressions. This is probably impossible, but I felt obligated to mention it. This is impossible, axe-core is causing one of the violations. It's worth mentioning that the axe extension also fails on this page.
• Disable the CSP (Page.setBypassCSP) for the afterPass/getArtifact phase of gathering. This is an extreme measure, but it might prevent future CSP issues. I couldn't get this to work, I think the function doesn't allow us to bypass sandboxing.

Page.getAppManifest: I haven't narrowed down the cause of this one yet. I'm hoping it is caused by a prior Page.evaluate error. Page.getAppManifest timeout went away once I eliminated/skipped the prior Page.evaluate timeouts.

[adamraine]

I filed upstream about this: https://bugs.chromium.org/p/chromium/issues/detail?id=1222763

[Seirdy]

Also impacts web.dev and WebPageTest's optional Lighthouse audit. Does not seem to break PSI or, surprisingly, GTMetrix; GTMetrix claims to use Lighthouse 7.4.0 (sample audit of a page whose sandbox breaks vanilla Lighthouse).

My current workaround for this (and, before it was fixed, #4386) was to just offer a subdomain with a less restrictive CSP for tests.