ignoreHTTPSErrors is not working when request interception is on #1159

Khady · 2017-10-25T13:01:53Z

Steps to reproduce

Tell us about your environment:

Puppeteer version: ab9b34c
Platform / OS version: debian stretch, nodejs 8.8
URLs (if applicable): https://halva.khady.info

What steps will reproduce the problem?

Just launch this code to see the problem:

// https.js
'use strict';

const puppeteer = require('puppeteer');

const URL = "https://halva.khady.info/";

(async() => {
  const args = [
    "--disable-setuid-sandbox",
    "--no-sandbox",
  ];
  const options = {
    args,
    headless: true,
    ignoreHTTPSErrors: true,
  };
  const browser = await puppeteer.launch(options);
  const page = await browser.newPage();
  await page.setRequestInterception(true);
  page.on("request", (request) => {
    if (request.resourceType === "Image") {
      request.abort();
    } else {
      request.continue();
    }
  });
  await page.goto(URL, { timeout: 8000, waitUntil: "load" });
  const html = await page.content();
  console.log(html);
  await page.close();
  await browser.close();
})();

What is the expected result?

With the request interception disabled:

  // await page.setRequestInterception(true);
  // page.on("request", (request) => {
  //   if (request.resourceType === "Image") {
  //     request.abort();
  //   } else {
  //     request.continue();
  //   }
  // });

$ nodejs misc/https_headless.js 
<html><head><title>potkw</title></head>
<body>
Khady Khady roule roule

</body></html>

What happens instead?

$ nodejs misc/https_headless.js 
(node:23556) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 1): Error: Navigation Timeout Exceeded: 8000ms exceeded
(node:23556) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
^C

The text was updated successfully, but these errors were encountered:

ali-habibzadeh · 2017-11-06T10:01:44Z

Same issue here

Khady · 2017-11-10T09:23:15Z

If you can use puppeteer without chrome headless, a solution seems to be the --ignore-certificate-errors flags. Unfortunately it doesn't work with chrome headless.

Khady · 2017-11-10T09:40:06Z

And actually the problem is here even if ignoreHTTPSErrors is set to false:

'use strict';

const puppeteer = require('puppeteer');

const URL = "https://halva.khady.info/";

(async () => {
  const args = [
    "--disable-setuid-sandbox",
    "--no-sandbox",
  ];
  const options = {
    args,
    headless: true,
    ignoreHTTPSErrors: false,
  };
  const browser = await puppeteer.launch(options);
  const page = await browser.newPage();
  await page.setRequestInterception(true);
  page.on("request", (request) => {
    if (request.resourceType === "image") {
      request.abort();
    } else {
      request.continue();
    }
  });
  await page.goto(URL, { timeout: 8000, waitUntil: "load" });
  const html = await page.content();
  console.log(html);
  await page.close();
  await browser.close();
})();

$ nodejs bhttps.js 
(node:321) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 1): Error: Navigation Timeout Exceeded: 8000ms exceeded
(node:321) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
^C

vc1 · 2017-11-21T15:48:56Z

#1441 same as yours

use this

args: [
    '--ignore-certificate-errors',
    '--ignore-certificate-errors-spki-list '
]

Khady · 2017-11-22T02:58:26Z

This work only when you launch puppeteer with headless: false. See #1159 (comment)

BTW, the --ignore-certificate-errors-spki-list flag is not doing anything if you don't use the --user-data-dir flag according to this documentation.

boligolov · 2017-12-20T12:59:29Z

I have '--ignore-certificate-errors' in arguments and ignoreHTTPSErrors is TRUE, but receive ERR_CERT_AUTHORITY_INVALID error from Chromium. Url for example: 'https://ostin.com/ru/ru/'

jsteel · 2017-12-22T16:42:29Z

I have all those ignore flags on and interception on. I only get an ERR_CERT_AUTHORITY_INVALID when running the script against the box I'm on if it uses a self signed cert. If I hit the same box from another server, it works fine.

aslushnikov · 2018-01-12T02:31:37Z

Filed upstream: crbug.com/801426

This however does not resolve a current issue where chrome headless hangs in self signed ceritificate errors, even with these flags. It only works with `headless: false`, but should hopefully be fixed soon: puppeteer/puppeteer#1159 (comment)

maZahaca · 2018-03-13T04:10:01Z

Is there any progress?

kevinmu · 2018-03-23T02:52:34Z

I ran into the same problem, worked around it by doing the following:

const puppeteer = require('puppeteer');

async function visit() {
    const browser = await puppeteer.launch({
        args: [
            '--disable-setuid-sandbox',
            '--no-sandbox',
            '--ignore-certificate-errors',
        ],
        ignoreHTTPSErrors: true,
        headless: true,
    });

    // this initial visit is meant to bypass the certificate errors; don't intercept anything yet.
    var url = "SOME_URL_ON_THE_SAME_DOMAIN.com"
    const page = await browser.newPage();
    await page.setViewport({width: 1300, height: 1000});
    await page.goto(url); 

    // intercept requests; at this point we've already bypassed the certificate errors,
    // so the subsequent page visit will load.
    await page.setRequestInterception(true);
    await page.on('request', interceptedRequest => {
        console.log(interceptedRequest.url());
        interceptedRequest.continue();
    });

    // visit the actual page you want to do stuff on.
    var actualURL = "SOME_URL_ON_THE_SAME_DOMAIN.com/ACTUAL_PAGE"
    await page.goto(actualURL, {waitUntil: 'networkidle0'});
    await browser.close();
}

However, even with this work-around, I still think that the fix for this issue is super high priority. Thanks!

leem32 · 2018-04-05T17:10:57Z

This is a major problemo. Hope it gets fixed soon.

wrightm · 2018-04-19T08:11:27Z

Any update on this front?

Looks like there is a solution using the chrome driver https://bugs.chromium.org/p/chromium/issues/detail?id=721739 and chrome 65/66 .

Capybara.register_driver :headless_chromium do |app|
      capabilities = Selenium::WebDriver::Remote::Capabilities.chrome(
        acceptInsecureCerts: true,
        binary: '/usr/bin/chromium-browser',
        chromeOptions: {
          'args' => ['--headless', '--disable-web-security', '--incognito',
                     '--no-sandbox', '--disable-gpu', '--window-size=1920,1080']
        })
      Capybara::Selenium::Driver.new(
        app,
        browser: :chrome,
        desired_capabilities: capabilities
      )
    end

I'll try and find sometime to figure how to apply this to puppeteer/chrome dev tools. If anyone gets there before me please post it up.

c094728 · 2018-04-25T05:13:08Z

I'm having the same problem and the work-around given by kevinmu didn't work for me. It will run fine running on my linux virtualbox machine but will give a time-out running in a docker container.
I'm using a node:9 docker image. If I remove the {waitUntil: ['networkidle0']} it will run and not timeout on docker but then when I retrieve all the anchors from the page, most are missing because the ajax calls have not yet finished updating the page. I can put in a timed wait but that is not reliable

vincentlong889 · 2018-05-28T06:32:23Z

Is there any progress?

aslushnikov · 2018-06-14T22:53:28Z

This bug will be fixed once Chromium completes migration onto Network Service.

As a workaround, network service could be enabled via a runtime flag --enable-feature=NetworkService.

Beware! NetworkService is not completed yet and doesn't pass all pptr tests.

Still, the following works for me:

const puppeteer = require('puppeteer');

const URL = "https://halva.khady.info/";

(async() => {
  const browser = await puppeteer.launch({
    args: ['--enable-features=NetworkService'],
    headless: true,
    ignoreHTTPSErrors: true,
  });
  const page = await browser.newPage();
  await page.setRequestInterception(true);
  page.on("request", (request) => {
    if (request.resourceType === "Image")
      request.abort();
    else
      request.continue();
  });
  await page.goto(URL, { timeout: 8000, waitUntil: "load" });
  const html = await page.content();
  console.log(html);
  await page.close();
  await browser.close();
})();

vincentlong889 · 2018-06-15T05:24:20Z

When I use this args: --enable-feature=NetworkService
then --proxy-server=xxx is not working!

seems like I use '--no-sandbox'，and proxy is not working,
https://groups.google.com/a/chromium.org/forum/#!topic/network-service-dev/qr7yCQeoT4o

browser = await puppeteer.launch({
      ignoreHTTPSErrors: true,
      args: [
        '--no-sandbox',
        '--disable-dev-shm-usage',
        '--enable-features=NetworkService',
        '--proxy-server=' + proxyUrl
      ]
    });

garris · 2018-06-15T06:10:14Z

@aslushnikov thanks for he info — I will give this a try tomorrow. Good to know a fix is on the roadmap!

fudali · 2018-07-03T14:26:09Z

@aslushnikov

Thanks for the workaround with --enable-feature=NetworkService
It works for me, but I also can't use proxy now

vincentlong889 · 2018-07-05T08:05:30Z

use google-chrome-unstable instead of puppeteer's chromium, setRequestInterception is working

const browser = await puppeteer.launch({
      executablePath: 'google-chrome-unstable',
      ignoreHTTPSErrors: true,
      args: ['--disable-dev-shm-usage']
    });

bluepeter · 2018-07-30T21:52:29Z

Side effect of using --enable-feature=NetworkService is that browser caching no longer appears to work.

leem32 · 2018-08-03T12:29:22Z

Just tried --enable-feature=NetworkService with Puppeteer 1.6.2 and accompanying Chromium, but it's still not working.

I tested using the URL http://finishline.com which hangs when using with await page.setRequestInterception(true); and there was no difference using --enable-feature=NetworkService as without it. Puppeteer just hangs up until goto timeout ends,

browser launch:
browser = await puppeteer.launch({ args: ['--enable-features=NetworkService'], headless: true, timeout: 30000, ignoreHTTPSErrors: true });

goto:
await page.goto(url, { timeout: 60000, waitUntil: 'load' })

fudali · 2018-08-21T09:51:47Z

Why is it closed? Isn't it gonna be resolved?

ntzm · 2018-08-29T11:21:24Z

NetworkService seems to break setting and getting cookies as well

maZahaca · 2018-09-03T18:54:55Z

Wow Wow Wow. The original issue from Chromium was just fixed! Yay!

Let's get this stuff moving...

This roll includes: - https://crrev.com/588420 - DevTools: teach request interception to work with Security.setIgnoreCertificateErrors Fixes puppeteer#1159.

maZahaca · 2018-09-04T06:48:22Z

Hi @aslushnikov, any ETA for this to be released?

This roll includes: - https://crrev.com/588420 - DevTools: teach request interception to work with Security.setIgnoreCertificateErrors Fixes #1159.

aslushnikov · 2018-09-04T09:10:02Z

@maZahaca: pptr 1.8.0 will be released on September, 6.

vincentlong889 · 2018-09-04T14:26:32Z

Great job!

bluepeter · 2018-09-18T03:10:40Z

Note: this is still an issue for those of us trying to run on Linux. See: https://bugs.chromium.org/p/chromium/issues/detail?id=877075

alexandrzavalii · 2019-03-17T12:46:49Z

as @bluepeter mentioned its still an issue.
Its not working on google cloud which is Linux based
Any workarounds?

webhype · 2019-05-26T13:27:37Z

Why is this issue being "aggressively closed?" Nobody there uses Mac OS? Not one person tested this on Unix-style systems?

Not one of the above examples, with or without any special args[], works on Mac OS, not for secure sites nor for https sites. Puppeteer 1.17 & Chrome 76

aslushnikov · 2019-06-02T22:19:01Z

Why is this issue being "aggressively closed?" Nobody there uses Mac OS? Not one person tested this on Unix-style systems?

@webhype The core issue was fixed and we added a test to make sure it doesn't regress. We run all our tests continuously on Mac, Linux and Windows - so yes, we fix across the variety of environments, including UNIX-style systems.

Not one of the above examples, with or without any special args[], works on Mac OS, not for secure sites nor for https sites. Puppeteer 1.17 & Chrome 76

The https://halva.khady.info/ is down - so the example doesn't work any more. I just tried with https://expired.badssl.com/ instead and it worked.

If something doesn't work for you - please file a new issue with a script that reproduces the problem. We'll be happy to help!

rodoabad · 2019-06-12T22:06:37Z

@aslushnikov puppeteer-firefox has this issue still. Is there a separate thread for it?

joelharkes · 2019-07-02T15:40:31Z

Seems to still be an issue setRequestInterception(true) in docker as well when using the following setup:

  var executablePath = inDocker ?  : undefined;
  return launch({
    headless: true,
    ["--no-sandbox", "--disable-setuid-sandbox", "--disable-dev-shm-usage"] ,
    "/usr/bin/chromium-browser",
    defaultViewport: { width: 1000, height: 1600 },
    // dumpio: false,
    // ignoreHTTPSErrors: true
  });

daddydrac · 2020-03-25T05:13:18Z

I am having same problem with https site, while running puppeteer in docker

javierfuentesm · 2020-06-26T14:00:02Z

@aslushnikov puppeteer-firefox has this issue still. Is there a separate thread for it?

Did you find a solution? I still have the same problem with firefox

JohnDotOwl · 2020-09-17T10:25:09Z

What the shit, after so many bloody years, still the same issue.

ghost mentioned this issue Nov 16, 2017

fix(Browser): fix Browser.create method - options mapping #1403

Merged

aslushnikov added bug upstream labels Jan 12, 2018

This was referenced Jan 12, 2018

setRequestInterception + ignoreHTTPSErrors , Unable to open the site certificate error #1441

Closed

Hanging intercepted request #759

Closed

aslushnikov added the P1 label Jan 12, 2018

aslushnikov mentioned this issue Jan 12, 2018

Page.goto may hang until timeout when setRequestInterceptionEnabled is true #1135

Closed

HanXHX mentioned this issue Jan 22, 2018

NFR: force IP on page.goto #1831

Closed

leem32 mentioned this issue Apr 5, 2018

page.setRequestInterception(true); causes invalid SSL pages to hangup and timeout instead of triggering an error. #2216

Closed

yujiosaka mentioned this issue Apr 8, 2018

Question: Extending/Customizing the behavior of crawler yujiosaka/headless-chrome-crawler#204

Closed

aslushnikov mentioned this issue Aug 7, 2018

RequestInterception: request.continue does nothing when site requires client certificate #3028

Closed

aslushnikov mentioned this issue Aug 31, 2018

page.goto never resolves when page.setRequestInterception is used (for some urls) #3118

Closed

aslushnikov mentioned this issue Sep 3, 2018

feat(chromium): roll Chromium to r588429 #3181

Merged

aslushnikov closed this as completed in #3181 Sep 4, 2018

aslushnikov added a commit that referenced this issue Sep 4, 2018

feat(chromium): roll Chromium to r588429 (#3181)

32c0363

This roll includes: - https://crrev.com/588420 - DevTools: teach request interception to work with Security.setIgnoreCertificateErrors Fixes #1159.

aslushnikov mentioned this issue Sep 4, 2018

'ignoreHTTPSErrors' option is not working well. #3119

Closed

glennjones mentioned this issue Sep 19, 2018

Puppeteer v1.8.0 browserless/browserless#63

Closed

rymdhund mentioned this issue Sep 26, 2018

hang with request interception + ignoreHTTPSErrors miyakogi/pyppeteer#150

Closed

jlcd mentioned this issue Oct 19, 2018

page.goto() with a certain URL throws TimeoutException when request interception is enabled, even if no request is aborted #3384

Closed

nqzhang mentioned this issue Mar 13, 2019

puppeteer 某些页面无法完成加载 binux/pyspider#879

Closed

bi-kai mentioned this issue Oct 1, 2020

how to deal with "net::ERR_CERT_AUTHORITY_INVALID" with wrong certificate ? browserless/browserless#1004

Closed

dgtlmoon mentioned this issue Dec 16, 2022

IgnoreHTTPSErrors only sometimes works? browserless/browserless#2485

Closed

ignoreHTTPSErrors is not working when request interception is on #1159

ignoreHTTPSErrors is not working when request interception is on #1159

Comments

Khady commented Oct 25, 2017

Steps to reproduce

ali-habibzadeh commented Nov 6, 2017

Khady commented Nov 10, 2017

Khady commented Nov 10, 2017

vc1 commented Nov 21, 2017 • edited

Khady commented Nov 22, 2017

boligolov commented Dec 20, 2017 • edited

jsteel commented Dec 22, 2017 • edited

aslushnikov commented Jan 12, 2018

maZahaca commented Mar 13, 2018

kevinmu commented Mar 23, 2018 • edited

leem32 commented Apr 5, 2018 • edited

wrightm commented Apr 19, 2018

c094728 commented Apr 25, 2018

vincentlong889 commented May 28, 2018

aslushnikov commented Jun 14, 2018

vincentlong889 commented Jun 15, 2018 • edited

garris commented Jun 15, 2018

fudali commented Jul 3, 2018

vincentlong889 commented Jul 5, 2018

bluepeter commented Jul 30, 2018

leem32 commented Aug 3, 2018 • edited

fudali commented Aug 21, 2018

ntzm commented Aug 29, 2018

maZahaca commented Sep 3, 2018 • edited

maZahaca commented Sep 4, 2018

aslushnikov commented Sep 4, 2018

vincentlong889 commented Sep 4, 2018

bluepeter commented Sep 18, 2018

alexandrzavalii commented Mar 17, 2019 • edited

webhype commented May 26, 2019

aslushnikov commented Jun 2, 2019 • edited

rodoabad commented Jun 12, 2019

joelharkes commented Jul 2, 2019

daddydrac commented Mar 25, 2020

javierfuentesm commented Jun 26, 2020

JohnDotOwl commented Sep 17, 2020

vc1 commented Nov 21, 2017 •

edited

boligolov commented Dec 20, 2017 •

edited

jsteel commented Dec 22, 2017 •

edited

kevinmu commented Mar 23, 2018 •

edited

leem32 commented Apr 5, 2018 •

edited

vincentlong889 commented Jun 15, 2018 •

edited

leem32 commented Aug 3, 2018 •

edited

maZahaca commented Sep 3, 2018 •

edited

alexandrzavalii commented Mar 17, 2019 •

edited

aslushnikov commented Jun 2, 2019 •

edited