page. setRequestInterception causes network requests to hang/timeout

[joelgriffith]

Steps to reproduce

Tell us about your environment:

• Puppeteer version: 1.20.0
• Platform / OS version: macOS (10.14.5 (18F203))
• URLs (if applicable): https://www.mapbox.com/gallery/#ice-cream
• Node.js version: 10.15.2

What steps will reproduce the problem?
Setting request interception causes certain requests to never completely resolve (though the goto method doesn't seem to mind).

Try the following:

const puppeteer = require('puppeteer');

async function run() {
  let browser = null;

  try {
    browser = await puppeteer.launch();

    const page = await browser.newPage();
    await page.goto('https://www.mapbox.com/gallery/#ice-cream', { waitUntil: 'networkidle0' });
    await page.screenshot({ path : './temp.png' });
  } catch (e) {
    console.error(`Saw error:`, e);
  } finally {
    if (browser) {
      browser.close();
    }
  }
}

run();

This returns a screenshot of:

Now, turn on request interception:

const puppeteer = require('puppeteer');

async function run() {
  let browser = null;

  try {
    browser = await puppeteer.launch();

    const page = await browser.newPage();
    await page.setRequestInterception(true);
    page.on('request', r => r.continue());
    await page.goto('https://www.mapbox.com/gallery/#ice-cream', { waitUntil: 'networkidle0' });
    await page.screenshot({ path : './temp.png' });
  } catch (e) {
    console.error(`Saw error:`, e);
  } finally {
    if (browser) {
      browser.close();
    }
  }
}

run();

This produces the following, as mapping tiles don't load (they hang)

What is the expected result?
All network requests should resolve, when the condition matches

What happens instead?
Network requests never resolve completely.

My suspicion is that this has to do with the version of Chromium bundled with pptr, but I haven't checked prior version just yet.

[dege88]

I got the same issue on different urls. Reverting to a different puppeteer version solved the issue.

[joelgriffith]

@dege88 what version was that, out of curiosity?

[laoshu133]

Rollback to 1.9.0 is available.

[arydjmal]

@joelgriffith did you have any luck with this? Having the same problem...

[themaues]

Hello,
any ideas here? having same issue.
Thx

[joelgriffith]

I've tried rolling back to previous versions, however the issue still seems present. My guess is that it's something embedded into Chrome and not how puppeteer is interacting with the Chrome Devtools Protocol

[airhorns]

Are y'all still experiencing this with puppeteer 2.0.0 or 2.1.0 ?

[dpereira411]

yes! on Mac I have this behaviour

[cjroebuck]

Seems to be an issue with blob: protocol url's

[geiserman]

We are using mapbox as map provider
The map is not loaded if using interceptor, similar to @joelgriffith's issue described above
Map requests get the warning message

this is my interceptor code:

    await page.setRequestInterception(true);

    page.on('request', (interceptedRequest) => {
        if (DOMAINS_BLACKLIST.includes(new URL(interceptedRequest.url()).host)) {
            logger.info(`Blocked url:${interceptedRequest.url()}`);
            requestsBlocked.push(interceptedRequest.url());
            interceptedRequest.abort();
        } else {
            logger.silly(`Intercepted request continue:${interceptedRequest.url()}`);
            interceptedRequest.continue();
        }
        interceptedRequest.continue();
    });

I tried with Puppeteer 1.19.0 and
Puppeteer 2.1.1
No difference between those versions

[undefinedfunction]

Not sure if this will help anybody that's stuck but I had this 'hanging' issue and found I wasn't calling continue. When I call continue, with page.setRequestInterception(true), it doesn't hang.

Found this issue wanted to share in case it helps someone.

On Win 10, Chromium Version 80.0.3987.0 (Developer Build) (64-bit) with following dependencies:

"dependencies": {
    "puppeteer": "^2.1.1",
    "puppeteer-extra": "^3.1.9",
    "puppeteer-extra-plugin-stealth": "^2.4.9",
    "sanitize-filename": "^1.6.3"
  }

This seems to work:

var stopRequest = (request) => {
  if(something){
    request.abort();
  }
  request.continue();
}
page = await browser.newPage()
await page.setRequestInterception(true);
await page.on('request', stopRequest);

[mstr-pezhang]

Tested with v2.1.1. Now the problem seems a bit different. Blob URLs are already loaded and there's no "CAUTION: request is not finished yet!" in Timing panel.

However, workers are still empty.

[rohitsg]

Enabling await page.setRequestInterception(true); stops futher execution, and removing await page.setRequestInterception(true); will run smoothly. There is issue with await page.setRequestInterception(true);

[rosstex]

Hey y'all, I found the solution to this. The download script for Chrome was pulling the latest nightly version, which is clearly full of bugs. I downloaded a stable binary of Chrome and symlinked it to the downloaded binary location in the ~/.local folder, and everything went smoothly.

[JohnDotOwl]

Facing the issue still.

[h8hawk]

Facing the issue in 2021 as well

[dragosMC91]

This is also happening for me, using puppeteer "version": "11.0.0"

[dragosMC91]

This is also happening for me, using puppeteer "version": "11.0.0"

I just noticed that if I abort .woff resources, it no longer hangs

[haxtordotid]

I recently faceing same issue my target URL is google and youtube, fb, instagram , but it's only working with instagram and facebook, when entering google/ youtube the page not loaded and raise timeout
I using Puppeeteer 12.0.1 & 11.0.0 and that the result

And when i try to downgrade version to 7.1.0
Everything is going well with all my target URL

[clouedoc]

Same problem with Google + puppeteer 11.0.0, despite using request.continue()
Rolling back to 7.1.0 does not fix the issue.
Other websites such as https://httpbin.org/ip work well.
I'm still investigating this issue, will report progress.

Reproduction code

  const browser = await puppeteer.launch({
    executablePath: '/Applications/Google Chrome.app/Contents/MacOS/Google Chrome',
    headless: false
  });

  const page = await browser.newPage();
  await page.setRequestInterception(true);
  page.on('request', async (req) => {
    console.log('received a requst!!!');
    await req.continue();
    console.log('told the request to continue ;)');
  });
  await page.goto('https://www.google.com');

Edit: hacky solution

const puppeteer = require('puppeteer');

async function main() {
  const browser = await puppeteer.launch({
    executablePath: '/Applications/Google Chrome.app/Contents/MacOS/Google Chrome',
    headless: false
  });

  const page = await browser.newPage();
  page.on('request', async (req) => { await req.continue(); });
  await page.setRequestInterception(true);
  await page.goto('https://google.com/' + Date.now())
  await page.click(`a[href*=google]`)
}

void main();

[abazlinton]

Just in case you make the same mistake as me

Make sure you set up -

page.on("request", (request) => ...);

Before -

await page.goto(...)

The docs state -

"Once request interception is enabled, every request will stall unless it's continued, responded or aborted."

So page.on("request", (request) => request.continue()) is mandatory for every request you want to continue.

[dkarlovi]

Check that you're not using Google Fonts on the page. Google Fonts (and, I'm assuming, other Google properties?) have an implicit rate limit. I've just debugged it to removing Google Fonts from my tested pages and the network timeout disappeared. They don't reject your request, they time out (at least at the time of this writing).

[stale]

We're marking this issue as unconfirmed because it has not had recent activity and we weren't able to confirm it yet. It will be closed if no further activity occurs within the next 30 days.

[OrKoN]

I believe there have been many fixes and a whole new cooperative interception mode landed since the issue was reported. If it still happens, please re-open it with a repro using the latest Puppeteer version.

[Shane32]

I'm having the same problem on both Puppeteer 10 and 17. Puppeteer 17 is the newest version that does not cause breaking changes in my application. Adding the following code causes most of my visual regression tests to fail:

await page.setRequestInterception(true);
await page.on('request', async request => {
    await request.continue();
});

All of the URLs the page requires (which include file:// and blob urls) can be monitored with console.log(request.url()), so it would seem that it is wired up correctly.

I'm using storybook.js 6.5.9 (and the storyshots addon) to perform visual regression tests against a Create React App website. See:

• https://storybook.js.org/addons/@storybook/addon-storyshots-puppeteer

I have not done further diagnosis to determine what could be causing the failure. I'm using React lazy loading, so that may impact results; not sure.

My workaround is to use this code:

// do not use setRequestInterception
await page.on('request', request => {
  // cause any external requests to fail
  if (request.url().startsWith('http')) {
    // log the url that attempted to load
    console.log('page tried loading: ' + request.url());
    // cause the test to fail
    throw 'error';
  }
});