Page with CSP breaks puppeteer

[aslushnikov]

A page with CSP will break some of puppeteer methods, such as page.addScriptTag. At the very least, we should fail gracefully.

It would be ideal to have a way to disable CSP altogether.

Reported in: #1091

[rakeshpai]

+1 for ability to disable CSP.

Use case: I'm using puppeteer as a post-build crawler to generate a static dump of my SPA. When crawling, I'd like to inline critical CSS, and add a CSP directive for the CSS text hash. While I'm at it, I want to change the API server from localhost:port to api.example.com. However, if my page already had a CSP meta tag in it (which is good to have during development so I can catch CSP violations early), puppeteer starts getting funky. Would love it if I can disable CSP altogether when crawling.

[sradu]

+1 for ability to disable CSP.

Thank you.

[yujiosaka]

In order to disable CSP, it's not a complete solution but adding nonce-<base64-value> to script tag will relax the security blocks. It only works for the site where the CSP headers or meta tags have the nonce part. It doesn't work for other sites, but in this way we don't have to change the default behavior or disable security blocks.

I'm working on that idea, but it will be probably difficult to completely disable CSP with current Chrome DevTool Protocol because I don't find APIs to intercept/modify response headers to do that yet.

[niieani]

Here's my workaround:

  import fetch from 'node-fetch'

  const requestInterceptor = async (request) => {
    const url = request.url()
    const requestHeaders = request.headers()
    const acceptHeader = requestHeaders.accept || ''
    if (url.includes('some-website-with-csp.com') && (acceptHeader.includes('text/html'))) {
      const cookiesList = await page.cookies(url)
      const cookies = cookiesList.map(cookie => `${cookie.name}=${cookie.value}`).join('; ')
      delete requestHeaders['x-devtools-emulate-network-conditions-client-id']
      if (requestHeaders.Cookie) {
        requestHeaders.cookie = requestHeaders.Cookie
        delete requestHeaders.Cookie
      }
      const theseHeaders = Object.assign({'cookie': cookies}, requestHeaders, {'accept-language': 'en-US,en'})

      const init = {
        body: request.postData(),
        headers: theseHeaders,
        method: request.method(),
        follow: 20,
      }
      const result = await fetch(
        url,
        init,
      )
      const resultHeaders = {}
      result.headers.forEach((value, name) => {
        if (name.toLowerCase() !== 'content-security-policy') {
          resultHeaders[name] = value
        } else {
          console.log('CSP', `omitting CSP`, {originalCSP: value})
        }
      })
      const buffer = await result.buffer()
      await request.respond({
        body: buffer,
        resultHeaders,
        status: result.status,
      })
    } else {
      request.continue();
    }
  }

  await page.setRequestInterception(true)
  page.on('request', requestInterceptor)

Related: #599

[marcelmattsson]

@niieani, I'm getting requestStory undefined. is it something I'm missing or where does that variable come from?

[niieani]

@marcelmattsson apologies, that was just my logger I forgot to remove before posting (I use storyboard for logging). Corrected the snippet above.

[marcelmattsson]

haha no worries, guess I'm doing something else wrong then since I'm still getting the EvalError even though I also see the console log telling me its omitting CSP.

[niieani]

@marcelmattsson perhaps some TypeScript leftovers were causing it (I'm assuming you're not using it)? I've removed all the types from my snippet now (I think so at least).

[marcelmattsson]

Was never CSP in the headers at all... was meta tags with CSP who was the culprit.

[XBeg9]

+1 CSP breaks my flow. :( no workarounds as for now.

[niieani]

Hi @XBeg9, doesn't my workaround work for you?

[garris]

Thanks @niieani — your solution inspired this simplified approach for a backstopjs usecase garris/BackstopJS@67801c2

@aslushnikov looking forward to #2324! 🎁

[niieani]

Great to hear it was useful @garris! Also looking forward to #2324 :)

[o0101]

Another workaround (without removing CSP from the entire page, which depending on security requirements of your use may leave the site too wide open) is to parse the CSP script-src directive, and pick a URL that is allowed.

Then in your injected script tag create a fake URL from the origin allowed by CSP, for example, if CSP script-src looks like:

script-src 'self' https://*.example.com

then inject a script tag with the following src:

<script src=https://random343489.example.com/my-fake-path/my-script-001.js></script>

then create a fake "virtual proxy" to intercept requests for:

https://random343489.example.com/

and keep an internal map of

{
  ['my-fake-path/my-script-001.js']: {
    headers: { ... },
    responseBody: '...'
 }
}

and serve those headers and body in response to the request you intercepted from your injected script tag.

[aespinoza96]

By setting the flag page.setBypassCSP to true work for me.
#2324