Certificates error using puppeteer

[Kalliser]

Hello,

I encountered a weird ssl problem using puppeeter 1.2.0 or 1.3.0 (inside a container).

Looking at this example (works on https://try-puppeteer.appspot.com/)

const browser = await puppeteer.launch({
  ignoreHTTPSErrors: true,
  headless: true
});

const page = await browser.newPage();


await Promise.all([
  page.waitForNavigation({timeout: 30000, waitUntil: 'networkidle2'}),
  page.goto('http://www.latabledarc.com/trianon-assiette-a-pain.html')
]);

await Promise.all([
  page.waitFor(5000),
  page.click('#product-addtocart-button')
]);

await Promise.all([
  page.waitForNavigation({timeout: 30000, waitUntil: 'networkidle2'}),
  page.goto('https://www.latabledarc.com/checkout/onepage/')
]);

console.log(await page.url());

await browser.close();

The problem is that I get the following error

Error running your code. Error: net::ERR_CERT_AUTHORITY_INVALID

Instead of to be allowed to navigate on the target url.

But if I try this in my browser everything is fine. The website has a valid ssl certificate.

On my local environment I get the following message using dumpio :

0414/115606.318905:ERROR:nss_ocsp.cc(597)] No URLRequestContext for NSS HTTP handler. host: cacerts.thawte.com
[0414/115606.318962:ERROR:cert_verify_proc_nss.cc(980)] CERT_PKIXVerifyCert for www.latabledarc.com failed err=-8179

And this seems not to be the only website wich is affected by this problem.

Any clue ?

[RuslanTT]

Have the same issue, with **************, anyone know how to fix it?

[evidana]

You can also set an args array inside the launch config

args: [ '--ignore-certificate-errors' ]

I have noticed that this only seems to work when headless if false.

[Kalliser]

args: [ '--ignore-certificate-errors' ] dosn't change anything (we are in headless).
For RuslanTT as fas as we know the certificate is not present in the container so even with curl it's impossible to connect to the website. By adding the certificate manually curl is working but not chrome this seems because chrome has his own directory for certificate, but we currently don't know where is the directory for chrome.
But for https://www.crediteurope.ch the problem seems simpler as the certificate is a 30 year lifetime certificate he is probably untrusted by chrome, see: https://security.googleblog.com/2018/03/distrust-of-symantec-pki-immediate.html

[kmicol]

Having the same issue, tried using Canary, Chromium, and chrome. https://www.bhphotovideo.com trying to do the checkout process. Issue appears to be that when puppeteer opens the page, and trying to checkout,

"The SSL certificate used to load resources from https://ds-aksb-a.akamaihd.net will be distrusted in M70. Once distrusted, users will be prevented from loading these resources. See https://g.co/chrome/symantecpkicerts for more information."

[jrivera2000tmt]

Anyone seen a workaround for this? We've been banging our heads around this issue for a long while...

[Kalliser]

yes chrome is missing the latest Thawte certificate (ThawteRSACA2018.crt) in his list of authorities it's why he refuse to get into some website.
This can be resolve by adding the missgin certificate manually. It's a little tricky but it's works :
First chrome must be run once or it will not works
/home/node_modules/puppeteer/.local-chromium/linux-549031/chrome-linux/chrome --headless --no-sandbox --dump-dom https://www.google.com

Copy the missing cert file in a dir (for me it's home/ThawteRSACA2018.crt)
then
run
cd /root/.pki/nssdb; certutil -d sql:. -A -t "C,," -n ThawteRSACA2018 -i /home/ThawteRSACA2018.crt
And you're done

[kyungjoongo]

certutil -d sql:. -A -t "C,," -n ThawteRSACA2018 -i /home/ThawteRSACA2018.crt
NOT WORK
ERROR --> certutil: could not add certificate to token or database: SEC_ERROR_ADDING_CERT: Error adding certificate to database.

[dpanic]

Temporary hack/workaround is to use xvfb while headless: True

xvfb-run --server-args='-screen 0, 1024x768x16' /usr/bin/timeout 30 /usr/bin/node script.js

[codeinvain]

{headless:true, args: [ '--enable-features=NetworkService ]} worked for me

[dpanic]

@codeinvain Which version of puppeteer?

[codeinvain]

@dpanic puppeteer 1.7.0

[andersju]

It appears that sites with incomplete certificate chains don't work in headless mode. Which is problematic, as plenty of sites have this issue (and work fine in most browsers). https://incomplete-chain.badssl.com/ can be used to check this. With Puppeteer 1.7.0, the following:

const puppeteer = require('puppeteer');

(async () => {
  const browser = await puppeteer.launch({headless: true});
  try {
    const page = await browser.newPage();
    await page.goto('https://incomplete-chain.badssl.com/');
    await page.screenshot({path: 'example.png'});
  } catch (err) {
    console.log(err.toString());
  }
  await browser.close();
})();

...results in Error: net::ERR_CERT_AUTHORITY_INVALID at https://incomplete-chain.badssl.com/.

With const browser = await puppeteer.launch({headless: false});, it works.

With const browser = await puppeteer.launch({headless: true, args: ['--enable-features=NetworkService']});, it doesn't work - same error as above.

[codeinvain]

@andersju running the code above (with or without --enable-feaures) generates a screenshot for incomplete-chain.badssl.com
on osx 10.13.6 / node v8.9.1 / puppeteer 1.7.0
hope it helps

[andersju]

@codeinvain Thanks! Yeah, seems like it might be an issue with headless Chromium on Linux (I'm on Ubuntu 18.04, Node 8.11.4) rather than Puppeteer, here's someone with the same issue: https://groups.google.com/a/chromium.org/forum/#!msg/headless-dev/CbVEqMc7Rlk/nElqclg1DgAJ

[dpanic]

@andersju I agree. But It is more convenient to track bug on Github than on Google Groups :)

[rbairwell]

I'm the person who started the Google Groups thread (as I was still trying to identify if it was a problem with our setup - I also tried posting in the Puppeteer and headless-chrome Slack channels, but no responses from there either).

On Ubuntu 14.04.5 LTS (Trust Tahr) using Google Chrome 68.0.3440.106 :
/usr/bin/google-chrome --headless --dump-dom https://www.example.com
produces the DOM as expected, but:
/usr/bin/google-chrome --headless --dump-dom https://incomplete-chain.badssl.com
fails with:

[0823/115256.905080:ERROR:gpu_process_transport_factory.cc(1016)] Lost UI shared context.
Fontconfig warning: "/etc/fonts/fonts.conf", line 146: blank doesn't take any effect anymore. please remove it from your fonts.conf
[0823/115257.126541:ERROR:nss_ocsp.cc(601)] No URLRequestContext for NSS HTTP handler. host: cacerts.digicert.com
[0823/115257.126667:ERROR:cert_verify_proc_nss.cc(981)] CERT_PKIXVerifyCert for incomplete-chain.badssl.com failed err=-8179
<html><head></head><body></body></html>

Running on Macos 10.13.6 (High Sierra) using Google Chrome 68.0.3440.106:
/Applications/Google\ Chrome.app/Contents/MacOS/Google\ Chrome --headless --dump-dom https://www.example.com
produces the DOM as expected AND
/Applications/Google\ Chrome.app/Contents/MacOS/Google\ Chrome --headless --dump-dom https://incomplete-chain.badssl.com
returns the DOM as expected.

It's definitely an OS dependent issue from my point of view and isn't a Puppeteer issue directly.

[added]
Launching Chrome with the arguments --ignore-certificate-errors --enable-features=NetworkService does appear to work - until you turn on request interception (which is needed to handle htauth/htpass username/password authentication or to block/inspect urls).

[rbairwell]

I've raised this as an issue on https://bugs.chromium.org/p/chromium/issues/detail?id=877075 : any confirmation/example cases - feel free to add.

[bluepeter]

Anyone have any workarounds? This has been killing me for months.

[dochne]

@bluepeter it looks like the fix has been made in Chromium, but I'd imagine it'll take several months to bubble through to non-development Chrome.

[mujuonly]

const puppeteer = require('puppeteer');
(async () => {
    const browser = await puppeteer.launch({ignoreHTTPSErrors: true, acceptInsecureCerts: true, args: ['--proxy-bypass-list=*', '--disable-gpu', '--disable-dev-shm-usage', '--disable-setuid-sandbox', '--no-first-run', '--no-sandbox', '--no-zygote', '--single-process', '--ignore-certificate-errors', '--ignore-certificate-errors-spki-list', '--enable-features=NetworkService']});
    const page = await browser.newPage();
    try {

        await page.goto('https://www.hostwpsolutions.com/', {waitUntil: 'networkidle2', timeout: 59000});
        const cookies = await page._client.send('Network.getAllCookies');
        JSON.stringify(cookies, null, 4);
    } catch (e) {
        console.log(e);
    }

    await browser.close();
})();

Error: net::ERR_SSL_VERSION_OR_CIPHER_MISMATCH at https://www.xxxxxxsolutions.com/

[stephenjesus]

await puppeteer.launch({ignoreHTTPSErrors: true});

[iAbhinav]

You can also set an args array inside the launch config

args: [ '--ignore-certificate-errors' ]

I have noticed that this only seems to work when headless if false.

This worked for me

I initialised the browser as

const browser = await puppeteer.launch({headless:true, args: [ '--ignore-certificate-errors' ]});

[javierfuentesm]

For me its not working
{
headless: true,
ignoreHTTPSErrors: true,
args: ["--ignore-certificate-errors"],
}
neither
{
headless: true,
args: ["--ignore-certificate-errors"],
}

[HansUXdev]

This technically worked for me but it fails the jest test says:

Async callback was not invoked within the 5000 ms timeout specified by jest.setTimeout.Timeout - Async callback was not invoked within the 5000 ms timeout specified by jest.setTimeout.Error:

The code I'm using:

const puppeteer = require('puppeteer')
const delaySecond = async (time) => await new Promise(res => setTimeout(res, 1000 * time));

describe('Open the website in Chrome, FireFox and Chrome-Edge', () => {
  test('Should open a website with FIREFOX', async () => {
    let FIREsettings = {
      headless: false,
      ignoreHTTPSErrors: true, 
      acceptInsecureCerts: true, 
      product: 'firefox',
      // executablePath:'/Applications/Firefox.app/Contents/MacOS/firefox',
      args: ['--proxy-bypass-list=*', '--disable-gpu', '--disable-dev-shm-usage', '--disable-setuid-sandbox', '--no-first-run', '--no-sandbox', '--no-zygote', '--single-process', '--ignore-certificate-errors', '--ignore-certificate-errors-spki-list', '--enable-features=NetworkService']
    }
    let browser = await puppeteer.launch(FIREsettings)
    let page = await browser.newPage()
    try {
      await page.goto("https://google.com")
      await page.screenshot({
        path: './Browsers/FIRE.png',
        fullPage: true
      })
      let img = await page.waitForSelector('#hplogo')
    } 
    catch (error) {
      console.error(`FIRE`,error)
    }
    await delaySecond(100)
    await browser.close();
    
  })
})

[OldChi]

You can also set an args array inside the launch config

args: [ '--ignore-certificate-errors' ]

I have noticed that this only seems to work when headless if false.

cool, that works for me. By the way, how do you get to know the argument --ignore-certificate-errors? I didn't find it in pyppeteer documentation.