Client certificate authentication on pages. #1319

konstantinblaesi · 2017-11-08T10:18:36Z

I've looked through open/closed issues and the puppeteer API documentation. Is an API planned/possible? Is there an alternative solution? Will puppeteer just use some global chrome profile + certificate store and use the client certificates installed on the host?

aslushnikov · 2017-11-23T04:48:40Z

I don't understand. Can you please elaborate more on what's the usecaes?

konstantinblaesi · 2017-11-23T10:35:20Z

@aslushnikov I mean websites with mandatory or optional authentication via client certificates. I think on windows google chrome uses the certificates from the certificate store, but wouldn't it make sense to have a chrome/puppeteer api for that similar to how it supports basic authentication?

konstantinblaesi · 2017-12-19T13:53:14Z

@aslushnikov This chromium headless bug probably need to be solved first.

aslushnikov · 2018-01-10T04:56:12Z

I mean websites with mandatory or optional authentication via client certificates.

@konstantinblaesi can you please give an example of such a website? Honestly, it looks like I have no idea about the topic.

jasonparallel · 2018-02-27T18:47:31Z

@aslushnikov This would support sites that use pki for authentication (vs user/pass or oauth). For testing work it would be helpful to be able to specify what cert chrome passes during the ssl handshake when the server requests a client certificate.

I can't think of a public site that uses pki for authentication. It is somewhat common for intranet sites.

dapriett · 2018-03-08T14:33:11Z

Here is how I got around client cert authentication, hope it helps others. Basically just intercept the request, then fire the request off yourself using your favorite httpclient lib, and repond to the intercepted request with the response info.

'use strict';

const puppeteer = require('puppeteer');
const request = require('request');
const fs = require('fs');

(async () => {
    const browser = await puppeteer.launch();
    let page = await browser.newPage();

    // Enable Request Interception
    await page.setRequestInterception(true);

    // Client cert files
    const cert = fs.readFileSync('/path/to/cert.crt.pem');
    const key = fs.readFileSync('/path/to/cert.key.pem');

    page.on('request', interceptedRequest => {
        // Intercept Request, pull out request options, add in client cert
        const options = {
            uri: interceptedRequest.url(),
            method: interceptedRequest.method(),
            headers: interceptedRequest.headers(),
            body: interceptedRequest.postData(),
            cert: cert,
            key: key
        };

        // Fire off the request manually (example is using using 'request' lib)
        request(options, function(err, resp, body) {
            // Abort interceptedRequest on error
            if (err) {
                console.error(`Unable to call ${options.uri}`, err);
                return interceptedRequest.abort('connectionrefused');
            }

            // Return retrieved response to interceptedRequest
            interceptedRequest.respond({
                status: resp.statusCode,
                contentType: resp.headers['content-type'],
                headers: resp.headers,
                body: body
            });
        });

    });

    await page.goto('https://client.badssl.com/');
    await browser.close();
})();

aslushnikov · 2018-05-31T09:00:08Z

Dupe of #540.

camjackson · 2018-10-12T10:33:13Z

So I know that 'thanks' and 'me too' comments are terrible, but I really just have to say THANK YOU to @dapriett for that solution! The entire internet seems to be filled with people asking how to get around the whole cert selection thing with puppeteer and various other browser automation libraries, and the above comment solves it beautifully! 👌

Note that request also supports sending the cert as a single PFX/PKCS12 file, optionally with a passphrase. You can see the docs here.

camjackson · 2018-10-18T07:23:00Z

One more update to the above solution by @dapriett - there is a bug with puppeteer where gzipped, intercepted responses are not decompressed correctly. See #1707.

This is easily fixed though - when calling request, just add gzip: true to the options. This will cause request to do the unzipping for you, so the browser will receive the uncompressed response. I suppose technically you should probably then strip the Content-Encoding: gzip header, but it hasn't caused any problems for me so far.

isakoala · 2019-01-31T10:15:11Z

Thanks @dapriett, this solves the authentication, but when routing through request, pages don't load properly for me, with or without certs. I haven't tried alternative http libraries yet. Using puppeteer@1.11.0 and request@2.88.0.

camjackson · 2019-02-10T06:13:09Z

@isakoala Looks like the stylesheet for the page isn't being loaded so you're just seeing raw, unstyled html. Or possible there are XHRs not loading either. I would just try adding a bunch of console logs within the request interception to see which requests are being made, which are getting responses back, etc, etc.

konstantinblaesi · 2019-02-10T20:11:10Z

FYI we've observed that when servers request certificate authentication from puppeteer/chromium it get's stuck when the request interception feature is enabled. We're basically hit by #3471

isakoala · 2019-02-11T11:18:15Z

@camjackson indeed most stylesheets, all images and some other script files aren't loading. In my case, the latter was fatal because all post login content on the site I'm scraping is loaded using said scripts. Thanks, @konstantinblaesi, I'll keep an eye on that.

alexandrzavalii · 2019-03-18T13:17:53Z

@dapriett sorry about noob questions,
where do I get this files from?

   // Client cert files
    const cert = fs.readFileSync('/path/to/cert.crt.pem');
    const key = fs.readFileSync('/path/to/cert.key.pem');

I only managed to download .crt file from broken website.

konstantinblaesi · 2019-03-18T13:35:18Z

This ticket is about a client authenticating to the webserver using certificates instead of credentials (e.g. basic auth). You or your organization will probably generate these files.

alexandrzavalii · 2019-03-18T13:52:36Z

@konstantinblaesi
Got it ! Thanks !
I don't own the website which has insecure ssl certificate.
However, locally puppeteer works okay in headless, But whenever I deploy it to google cloud, I am stuck at await page.goto('bad_website.com). If I use google.com it works.

Another weird thing, is that same code works on AWS lambda.

I know its really difficult to tell what's the issue,but any suggestions would be greatly appreciated.

konstantinblaesi · 2019-03-18T14:40:14Z

You can always tell puppeteer/chromium to ignore https errors by setting ignoreHTTPSErrors: true when launching the browser see
https://github.com/GoogleChrome/puppeteer/blob/v1.13.0/docs/api.md#puppeteerlaunchoptions
You should probably check which CA signed the certificate of the website you're trying to crawl and see whether that CA is installed on the system you're using puppeter on.

alexandrzavalii · 2019-03-18T16:39:37Z

@konstantinblaesi Thanks for your help!
Sorry I forgot to mention that I am using alreadyignoreHTTPSErrors: true.with puppeteer: 1.13.0 and I have no luck.

    const browser = await puppeteer.launch({
      headless: true,
      ignoreHTTPSErrors: true,
      args: [
        '--incognito',
        '--disable-gpu',
        '--disable-dev-shm-usage',
        '--disable-setuid-sandbox',
        '--no-first-run',
        '--no-sandbox',
        '--no-zygote',
        '--single-process', // <- this one doesn't works in Windows
        '--ignore-certificate-errors',
        '--ignore-certificate-errors-spki-list',
        '--user-data-dir',
        '--enable-features=NetworkService'
      ]
    });

Every time I run await page.goto('bad_website.com') I have timeout error TimeoutError: Navigation Timeout Exceeded: on google cloud functions

Not sure how to check what CA are installed on google cloud.

In documentation it is mentioned that page.goto might cause the timeout is exceeded during navigation. but its not saying what might cause it.

https://github.com/GoogleChrome/puppeteer/blob/v1.13.0/docs/api.md#pagegotourl-options

camjackson · 2019-03-19T00:36:03Z

Hi @alexandrzavalii, as @konstantinblaesi said, this issue is about client cert authentication, which is a method for logging into websites by using an SSL certificate, rather than a username and password. It's an approach sometimes used by corporates for managing internal employee identities, but almost never used on the regular public internet.

It sounds like your issue is more to do with SSL cert validation on a normal website, which is a completely different issue. Topic aside, github issues is really for bug and issue tracking rather than user support, so you might be better off asking a question on somewhere like stack overflow. Good luck! 🙂

badeball · 2019-04-23T10:46:38Z

Thanks for your intercept example, @dapriett. My use case isn't about certificate authentication specifically, but simply self-signed certificates created for the purpose of testing a webserver application (that cannot be server over non-https due to reasons).

It bothers me to no end that people seem to equate the problems of

Using a provided, trusted certificate to request a webpage
Ignoring all errors related to certificates

I obviously want my test to resemble how the software is used as closely as possible and I don't get that by disabling certificate verification.

Some initial experimentation with request interception seemed to indicate that it did interfere with cookies and websockets 😞

filozof6 · 2019-08-25T11:38:18Z

Here is how I got around client cert authentication, hope it helps others. Basically just intercept the request, then fire the request off yourself using your favorite httpclient lib, and repond to the intercepted request with the response info.

'use strict';

const puppeteer = require('puppeteer');
const request = require('request');
const fs = require('fs');

(async () => {
    const browser = await puppeteer.launch();
    let page = await browser.newPage();

    // Enable Request Interception
    await page.setRequestInterception(true);

    // Client cert files
    const cert = fs.readFileSync('/path/to/cert.crt.pem');
    const key = fs.readFileSync('/path/to/cert.key.pem');

    page.on('request', interceptedRequest => {
        // Intercept Request, pull out request options, add in client cert
        const options = {
            uri: interceptedRequest.url(),
            method: interceptedRequest.method(),
            headers: interceptedRequest.headers(),
            body: interceptedRequest.postData(),
            cert: cert,
            key: key
        };

        // Fire off the request manually (example is using using 'request' lib)
        request(options, function(err, resp, body) {
            // Abort interceptedRequest on error
            if (err) {
                console.error(`Unable to call ${options.uri}`, err);
                return interceptedRequest.abort('connectionrefused');
            }

            // Return retrieved response to interceptedRequest
            interceptedRequest.respond({
                status: resp.statusCode,
                contentType: resp.headers['content-type'],
                headers: resp.headers,
                body: body
            });
        });

    });

    await page.goto('https://client.badssl.com/');
    await browser.close();
})();

this does not work for me it seems tho I intercept the request the browser just reloads and repeat the same request all over again :(

dcr007 · 2020-04-24T09:35:01Z

@dapriett , @camjackson - Are you referring to a private key to validate the cer ?
const key = fs.readFileSync('/path/to/cert.key.pem');
how do I get this key ? .

konstantinblaesi · 2020-04-24T09:48:51Z

@dcr007 These are most likely public (cert) and private key (key) files of your client. They can be generated with openssl or similar tools.

dcr007 · 2020-04-24T16:55:29Z

@konstantinblaesi - thanks for responding !, I managed to export the public(cert) from chrome from the website and then executing openssl x509 -inform DER -in publicCertificate.cer -out mynewcert.pem -text
How do I do export the private key for the certificate from mac os ? I tried digging into my 'keychains' but couldn't figure out , any pointers will be highly appreciated.
I'm currently experiencing this error from chromium while running the test:

"Identity Provider could not process the authentication request received. Delete your browser cache and stored cookies, and restart your browser. If you still experience issues after doing this, please contact your administrator."

konstantinblaesi · 2020-04-24T17:01:12Z

@dcr007 unfortunately I cannot be of any further help, because I have no practical experience with puppeteer and client certificate auth. I would suggest asking via https://github.com/puppeteer/puppeteer#q-i-have-more-questions-where-do-i-ask and https://gitter.im/puppeteer-chat/Lobby

ldrahnik · 2022-03-11T12:17:32Z

My workaroud is use headfull puppeter running in docker without GUI using xvfb:

Docker

Install Google Chrome & xvfb

RUN wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | apt-key add - \    && echo "deb http://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google.list
RUN apt-get update
RUN apt-get install -yq gconf-service libasound2 libatk1.0-0 libc6 libcairo2 libcups2 libdbus-1-3 libexpat1 libfontconfig1 libgcc1 libgconf-2-4 libgdk-pixbuf2.0-0 libglib2.0-0 libgtk-3-0 libnspr4 libpango-1.0-0 libpangocairo-1.0-0 libstdc++6 libx11-6 libx11-xcb1 libxcb1 libxcomposite1 libxcursor1 libxdamage1 libxext6 libxfixes3 libxi6 libxrandr2 libxrender1 libxss1 libxtst6 ca-certificates fonts-liberation libappindicator1 libnss3 lsb-release xdg-utils wget x11vnc x11-xkb-utils xfonts-100dpi xfonts-75dpi xfonts-scalable xfonts-cyrillic x11-apps
RUN apt-get install -yq xvfb google-chrome-stable

Add SSL certificate to Google Chrome

RUN apt-get install -yq libnss3-tools
RUN mkdir -p $HOME/.pki/nssdb
RUN pk12util -d sql:$HOME/.pki/nssdb -i /tmp/cert.pfx -w /tmp/cert.pfx.pass

Set up autoselecting added SSL certificate in Google Chrome

{ "AutoSelectCertificateForUrls": ["{\"pattern\":\"*\",\"filter\":{\"ISSUER\":{\"CN\":\"XY name\"}}}"]}

mkdir -p /etc/opt/chrome/policies/managed
cp policy.json /etc/opt/chrome/policies/managed

Run nodejs part with rendering redirected to xvfb

CMD xvfb-run --setver-args='-screen 0 1024x768x24" node index.js

Nodejs

Install npm Puppeteer package

npm install puppeteer

Run Puppeteer

const browser = await puppeteer.launch({ executablePath: '/usr/bin/google-chrome-stable', headless: false, args: [' - no-sandbox',' - disable-setuid-sandbox'] });

Tested on render.com. Example project. More detailed Medium story

aallvi · 2023-11-30T19:04:59Z

Here is how I got around client cert authentication, hope it helps others. Basically just intercept the request, then fire the request off yourself using your favorite httpclient lib, and repond to the intercepted request with the response info.

'use strict';

const puppeteer = require('puppeteer');
const request = require('request');
const fs = require('fs');

(async () => {
    const browser = await puppeteer.launch();
    let page = await browser.newPage();

    // Enable Request Interception
    await page.setRequestInterception(true);

    // Client cert files
    const cert = fs.readFileSync('/path/to/cert.crt.pem');
    const key = fs.readFileSync('/path/to/cert.key.pem');

    page.on('request', interceptedRequest => {
        // Intercept Request, pull out request options, add in client cert
        const options = {
            uri: interceptedRequest.url(),
            method: interceptedRequest.method(),
            headers: interceptedRequest.headers(),
            body: interceptedRequest.postData(),
            cert: cert,
            key: key
        };

        // Fire off the request manually (example is using using 'request' lib)
        request(options, function(err, resp, body) {
            // Abort interceptedRequest on error
            if (err) {
                console.error(`Unable to call ${options.uri}`, err);
                return interceptedRequest.abort('connectionrefused');
            }

            // Return retrieved response to interceptedRequest
            interceptedRequest.respond({
                status: resp.statusCode,
                contentType: resp.headers['content-type'],
                headers: resp.headers,
                body: body
            });
        });

    });

    await page.goto('https://client.badssl.com/');
    await browser.close();
})();

this code is not working for me :( the page throw blank after accept the use of certificate with

page.on('dialog', async dialog => {
await dialog.accept();
});

igorovisk · 2024-05-10T18:27:49Z

When i try doing the interception, I get an error:
Error: Request is already handled!

aslushnikov closed this as completed May 31, 2018

camjackson mentioned this issue Oct 12, 2018

How do I get rid of popup for chromium certificates selection, can I set it programmatically ? #540

Open

azasypkin mentioned this issue Aug 15, 2019

Reporting should work with server.ssl.clientAuthentication set to required elastic/kibana#43309

Open

jtrinh27 mentioned this issue Jun 2, 2021

Client certificate authentication on pages dsheiko/puppetry#89

Open

Yanchi4425 mentioned this issue Mar 7, 2022

Upload a plugin to an environment that uses a client certificate. kintone/js-sdk#1392

Open

Client certificate authentication on pages. #1319

Client certificate authentication on pages. #1319

Comments

konstantinblaesi commented Nov 8, 2017

aslushnikov commented Nov 23, 2017

Uh oh!

konstantinblaesi commented Nov 23, 2017

Uh oh!

konstantinblaesi commented Dec 19, 2017

Uh oh!

aslushnikov commented Jan 10, 2018

Uh oh!

jasonparallel commented Feb 27, 2018

Uh oh!

dapriett commented Mar 8, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

aslushnikov commented May 31, 2018

Uh oh!

camjackson commented Oct 12, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

camjackson commented Oct 18, 2018

Uh oh!

isakoala commented Jan 31, 2019

Uh oh!

camjackson commented Feb 10, 2019

Uh oh!

konstantinblaesi commented Feb 10, 2019

Uh oh!

isakoala commented Feb 11, 2019

Uh oh!

alexandrzavalii commented Mar 18, 2019

Uh oh!

konstantinblaesi commented Mar 18, 2019

Uh oh!

alexandrzavalii commented Mar 18, 2019

Uh oh!

konstantinblaesi commented Mar 18, 2019

Uh oh!

alexandrzavalii commented Mar 18, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

camjackson commented Mar 19, 2019

Uh oh!

badeball commented Apr 23, 2019

Uh oh!

filozof6 commented Aug 25, 2019

Uh oh!

dcr007 commented Apr 24, 2020

Uh oh!

konstantinblaesi commented Apr 24, 2020

Uh oh!

dcr007 commented Apr 24, 2020

Uh oh!

konstantinblaesi commented Apr 24, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ldrahnik commented Mar 11, 2022

Docker

Nodejs

Uh oh!

aallvi commented Nov 30, 2023

Uh oh!

igorovisk commented May 10, 2024

Uh oh!

dapriett commented Mar 8, 2018 •

edited

Loading

camjackson commented Oct 12, 2018 •

edited

Loading

alexandrzavalii commented Mar 18, 2019 •

edited

Loading

konstantinblaesi commented Apr 24, 2020 •

edited

Loading