Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ignoreHTTPSErrors is not working when request interception is on #1159

Closed
Khady opened this issue Oct 25, 2017 · 33 comments

Comments

Projects
None yet
@Khady
Copy link

commented Oct 25, 2017

Steps to reproduce

Tell us about your environment:

What steps will reproduce the problem?

Just launch this code to see the problem:

// https.js
'use strict';

const puppeteer = require('puppeteer');

const URL = "https://halva.khady.info/";

(async() => {
  const args = [
    "--disable-setuid-sandbox",
    "--no-sandbox",
  ];
  const options = {
    args,
    headless: true,
    ignoreHTTPSErrors: true,
  };
  const browser = await puppeteer.launch(options);
  const page = await browser.newPage();
  await page.setRequestInterception(true);
  page.on("request", (request) => {
    if (request.resourceType === "Image") {
      request.abort();
    } else {
      request.continue();
    }
  });
  await page.goto(URL, { timeout: 8000, waitUntil: "load" });
  const html = await page.content();
  console.log(html);
  await page.close();
  await browser.close();
})();

What is the expected result?

With the request interception disabled:

  // await page.setRequestInterception(true);
  // page.on("request", (request) => {
  //   if (request.resourceType === "Image") {
  //     request.abort();
  //   } else {
  //     request.continue();
  //   }
  // });
$ nodejs misc/https_headless.js 
<html><head><title>potkw</title></head>
<body>
Khady Khady roule roule

</body></html>

What happens instead?

$ nodejs misc/https_headless.js 
(node:23556) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 1): Error: Navigation Timeout Exceeded: 8000ms exceeded
(node:23556) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
^C
@ali-habibzadeh

This comment has been minimized.

Copy link

commented Nov 6, 2017

Same issue here

@Khady

This comment has been minimized.

Copy link
Author

commented Nov 10, 2017

If you can use puppeteer without chrome headless, a solution seems to be the --ignore-certificate-errors flags. Unfortunately it doesn't work with chrome headless.

@Khady

This comment has been minimized.

Copy link
Author

commented Nov 10, 2017

And actually the problem is here even if ignoreHTTPSErrors is set to false:

'use strict';

const puppeteer = require('puppeteer');

const URL = "https://halva.khady.info/";

(async () => {
  const args = [
    "--disable-setuid-sandbox",
    "--no-sandbox",
  ];
  const options = {
    args,
    headless: true,
    ignoreHTTPSErrors: false,
  };
  const browser = await puppeteer.launch(options);
  const page = await browser.newPage();
  await page.setRequestInterception(true);
  page.on("request", (request) => {
    if (request.resourceType === "image") {
      request.abort();
    } else {
      request.continue();
    }
  });
  await page.goto(URL, { timeout: 8000, waitUntil: "load" });
  const html = await page.content();
  console.log(html);
  await page.close();
  await browser.close();
})();
$ nodejs bhttps.js 
(node:321) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 1): Error: Navigation Timeout Exceeded: 8000ms exceeded
(node:321) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
^C
@vc1

This comment has been minimized.

Copy link

commented Nov 21, 2017

#1441 same as yours

use this

args: [
    '--ignore-certificate-errors',
    '--ignore-certificate-errors-spki-list '
]
@Khady

This comment has been minimized.

Copy link
Author

commented Nov 22, 2017

This work only when you launch puppeteer with headless: false. See #1159 (comment)

BTW, the --ignore-certificate-errors-spki-list flag is not doing anything if you don't use the --user-data-dir flag according to this documentation.

@boligolov

This comment has been minimized.

Copy link

commented Dec 20, 2017

I have '--ignore-certificate-errors' in arguments and ignoreHTTPSErrors is TRUE, but receive ERR_CERT_AUTHORITY_INVALID error from Chromium. Url for example: 'https://ostin.com/ru/ru/'

@jsteel

This comment has been minimized.

Copy link

commented Dec 22, 2017

I have all those ignore flags on and interception on. I only get an ERR_CERT_AUTHORITY_INVALID when running the script against the box I'm on if it uses a self signed cert. If I hit the same box from another server, it works fine.

@aslushnikov

This comment has been minimized.

Copy link
Contributor

commented Jan 12, 2018

Filed upstream: crbug.com/801426

@aslushnikov aslushnikov added the P1 label Jan 12, 2018

pocketjoso added a commit to pocketjoso/penthouse that referenced this issue Mar 11, 2018

add explicit ignore certificate errors argument for puppeteer
This however does not resolve a current issue where chrome headless
hangs in self signed ceritificate errors, even with these flags.
It only works with `headless: false`, but should hopefully be fixed soon:

GoogleChrome/puppeteer#1159 (comment)
@maZahaca

This comment has been minimized.

Copy link

commented Mar 13, 2018

Is there any progress?

@kevinmu

This comment has been minimized.

Copy link

commented Mar 23, 2018

I ran into the same problem, worked around it by doing the following:

const puppeteer = require('puppeteer');

async function visit() {
    const browser = await puppeteer.launch({
        args: [
            '--disable-setuid-sandbox',
            '--no-sandbox',
            '--ignore-certificate-errors',
        ],
        ignoreHTTPSErrors: true,
        headless: true,
    });

    // this initial visit is meant to bypass the certificate errors; don't intercept anything yet.
    var url = "SOME_URL_ON_THE_SAME_DOMAIN.com"
    const page = await browser.newPage();
    await page.setViewport({width: 1300, height: 1000});
    await page.goto(url); 

    // intercept requests; at this point we've already bypassed the certificate errors,
    // so the subsequent page visit will load.
    await page.setRequestInterception(true);
    await page.on('request', interceptedRequest => {
        console.log(interceptedRequest.url());
        interceptedRequest.continue();
    });

    // visit the actual page you want to do stuff on.
    var actualURL = "SOME_URL_ON_THE_SAME_DOMAIN.com/ACTUAL_PAGE"
    await page.goto(actualURL, {waitUntil: 'networkidle0'});
    await browser.close();
}

However, even with this work-around, I still think that the fix for this issue is super high priority. Thanks!

@leem32

This comment has been minimized.

Copy link

commented Apr 5, 2018

This is a major problemo. Hope it gets fixed soon.

@wrightm

This comment has been minimized.

Copy link

commented Apr 19, 2018

Any update on this front?

Looks like there is a solution using the chrome driver https://bugs.chromium.org/p/chromium/issues/detail?id=721739 and chrome 65/66 .

Capybara.register_driver :headless_chromium do |app|
      capabilities = Selenium::WebDriver::Remote::Capabilities.chrome(
        acceptInsecureCerts: true,
        binary: '/usr/bin/chromium-browser',
        chromeOptions: {
          'args' => ['--headless', '--disable-web-security', '--incognito',
                     '--no-sandbox', '--disable-gpu', '--window-size=1920,1080']
        })
      Capybara::Selenium::Driver.new(
        app,
        browser: :chrome,
        desired_capabilities: capabilities
      )
    end

I'll try and find sometime to figure how to apply this to puppeteer/chrome dev tools. If anyone gets there before me please post it up.

@c094728

This comment has been minimized.

Copy link

commented Apr 25, 2018

I'm having the same problem and the work-around given by kevinmu didn't work for me. It will run fine running on my linux virtualbox machine but will give a time-out running in a docker container.
I'm using a node:9 docker image. If I remove the {waitUntil: ['networkidle0']} it will run and not timeout on docker but then when I retrieve all the anchors from the page, most are missing because the ajax calls have not yet finished updating the page. I can put in a timed wait but that is not reliable

@vincentlong889

This comment has been minimized.

Copy link

commented May 28, 2018

Is there any progress?

@aslushnikov

This comment has been minimized.

Copy link
Contributor

commented Jun 14, 2018

This bug will be fixed once Chromium completes migration onto Network Service.

As a workaround, network service could be enabled via a runtime flag --enable-feature=NetworkService.

Beware! NetworkService is not completed yet and doesn't pass all pptr tests.

Still, the following works for me:

const puppeteer = require('puppeteer');

const URL = "https://halva.khady.info/";

(async() => {
  const browser = await puppeteer.launch({
    args: ['--enable-features=NetworkService'],
    headless: true,
    ignoreHTTPSErrors: true,
  });
  const page = await browser.newPage();
  await page.setRequestInterception(true);
  page.on("request", (request) => {
    if (request.resourceType === "Image")
      request.abort();
    else
      request.continue();
  });
  await page.goto(URL, { timeout: 8000, waitUntil: "load" });
  const html = await page.content();
  console.log(html);
  await page.close();
  await browser.close();
})();
@vincentlong889

This comment has been minimized.

Copy link

commented Jun 15, 2018

When I use this args: --enable-feature=NetworkService
then --proxy-server=xxx is not working!

seems like I use '--no-sandbox',and proxy is not working,
https://groups.google.com/a/chromium.org/forum/#!topic/network-service-dev/qr7yCQeoT4o

browser = await puppeteer.launch({
      ignoreHTTPSErrors: true,
      args: [
        '--no-sandbox',
        '--disable-dev-shm-usage',
        '--enable-features=NetworkService',
        '--proxy-server=' + proxyUrl
      ]
    });
@garris

This comment has been minimized.

Copy link

commented Jun 15, 2018

@aslushnikov thanks for he info — I will give this a try tomorrow. Good to know a fix is on the roadmap!

@fudali

This comment has been minimized.

Copy link

commented Jul 3, 2018

@aslushnikov

Thanks for the workaround with --enable-feature=NetworkService
It works for me, but I also can't use proxy now

@vincentlong889

This comment has been minimized.

Copy link

commented Jul 5, 2018

use google-chrome-unstable instead of puppeteer's chromium, setRequestInterception is working

const browser = await puppeteer.launch({
      executablePath: 'google-chrome-unstable',
      ignoreHTTPSErrors: true,
      args: ['--disable-dev-shm-usage']
    });
@sangaline

This comment has been minimized.

Copy link

commented Jul 22, 2018

@vincentlong889 Does the ignoreHTTPSErrors option actually work when you do that? Request interception works fine when I try with Chrome 69.0.3493.3 dev, but navigating to a page with a bad certificate fails. Could you share the exact version you're using if it does work?

@bluepeter

This comment has been minimized.

Copy link

commented Jul 30, 2018

Side effect of using --enable-feature=NetworkService is that browser caching no longer appears to work.

@leem32

This comment has been minimized.

Copy link

commented Aug 3, 2018

Just tried --enable-feature=NetworkService with Puppeteer 1.6.2 and accompanying Chromium, but it's still not working.

I tested using the URL http://finishline.com which hangs when using with await page.setRequestInterception(true); and there was no difference using --enable-feature=NetworkService as without it. Puppeteer just hangs up until goto timeout ends,

browser launch:
browser = await puppeteer.launch({ args: ['--enable-features=NetworkService'], headless: true, timeout: 30000, ignoreHTTPSErrors: true });

goto:
await page.goto(url, { timeout: 60000, waitUntil: 'load' })

@fudali

This comment has been minimized.

Copy link

commented Aug 21, 2018

Why is it closed? Isn't it gonna be resolved?

@ntzm

This comment has been minimized.

Copy link
Contributor

commented Aug 29, 2018

NetworkService seems to break setting and getting cookies as well

@maZahaca

This comment has been minimized.

Copy link

commented Sep 3, 2018

Wow Wow Wow. The original issue from Chromium was just fixed! Yay!

Let's get this stuff moving...

aslushnikov added a commit to aslushnikov/puppeteer that referenced this issue Sep 3, 2018

feat(chromium): roll Chromium to r588429
This roll includes:
- https://crrev.com/588420 - DevTools: teach request interception to work with
  Security.setIgnoreCertificateErrors

Fixes GoogleChrome#1159.
@maZahaca

This comment has been minimized.

Copy link

commented Sep 4, 2018

Hi @aslushnikov, any ETA for this to be released?

aslushnikov added a commit that referenced this issue Sep 4, 2018

feat(chromium): roll Chromium to r588429 (#3181)
This roll includes:
- https://crrev.com/588420 - DevTools: teach request interception to work with
  Security.setIgnoreCertificateErrors

Fixes #1159.
@aslushnikov

This comment has been minimized.

Copy link
Contributor

commented Sep 4, 2018

@maZahaca: pptr 1.8.0 will be released on September, 6.

@vincentlong889

This comment has been minimized.

Copy link

commented Sep 4, 2018

Great job!

@bluepeter

This comment has been minimized.

Copy link

commented Sep 18, 2018

Note: this is still an issue for those of us trying to run on Linux. See: https://bugs.chromium.org/p/chromium/issues/detail?id=877075

@alexandrzavalii

This comment has been minimized.

Copy link

commented Mar 17, 2019

as @bluepeter mentioned its still an issue.
Its not working on google cloud which is Linux based
Any workarounds?

LarenDorr added a commit to LarenDorr/puppeteer that referenced this issue Mar 19, 2019

feat(chromium): roll Chromium to r588429 (GoogleChrome#3181)
This roll includes:
- https://crrev.com/588420 - DevTools: teach request interception to work with
  Security.setIgnoreCertificateErrors

Fixes GoogleChrome#1159.
@webhype

This comment has been minimized.

Copy link

commented May 26, 2019

Why is this issue being "aggressively closed?" Nobody there uses Mac OS? Not one person tested this on Unix-style systems?

Not one of the above examples, with or without any special args[], works on Mac OS, not for secure sites nor for https sites. Puppeteer 1.17 & Chrome 76

@aslushnikov

This comment has been minimized.

Copy link
Contributor

commented Jun 2, 2019

Why is this issue being "aggressively closed?" Nobody there uses Mac OS? Not one person tested this on Unix-style systems?

@webhype The core issue was fixed and we added a test to make sure it doesn't regress. We run all our tests continuously on Mac, Linux and Windows - so yes, we fix across the variety of environments, including UNIX-style systems.

Not one of the above examples, with or without any special args[], works on Mac OS, not for secure sites nor for https sites. Puppeteer 1.17 & Chrome 76

The https://halva.khady.info/ is down - so the example doesn't work any more. I just tried with https://expired.badssl.com/ instead and it worked.

If something doesn't work for you - please file a new issue with a script that reproduces the problem. We'll be happy to help!

@rodoabad

This comment has been minimized.

Copy link

commented Jun 12, 2019

@aslushnikov puppeteer-firefox has this issue still. Is there a separate thread for it?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.