Page.Content freezes without error after running 101 times

[JohnDotOwl]

• Puppeteer version: puppeteer@1.12.1
• Platform / OS version: Windows 10 (20 Core , 64GB Ram)
• Node.js version: v10.15.0

What steps will reproduce the problem?
Run the script and wait to i == 1129 (around 101 page visit) it would just freeze and hang

Please include code that reproduces the issue.
Link to Full Code : https://gist.github.com/rainbowhat/00ee308129fd52b87156b5e0715f8fff

What is the expected result?
It's expected to keep running until 10,000

What happens instead?
It freezes and no errors appear, The output during the freeze is

After Page.Content
Hit End of Loop
Hit After Visit
1127
Before Page.content

I have to do a ctrl + c to end the program.

[vsemozhetbyt]

I can reproduce with this reduced code:

'use strict';

const puppeteer = require('puppeteer');

(async () => {
  const browser = await puppeteer.launch();
  const [page] = await browser.pages();

  for (let i = 0; i < 200; i++) {
    await page.goto('http://www.example.com/?page=' + i);
    console.log('Before Page.content', i);
    const html = await page.content();
    console.log('After Page.Content', i);
    await page.waitFor(500);
  }

  await browser.close();
})();

It stops at:

Before Page.content 100

[mishaberezin]

Have a similar issue with page.waitForSelector. At the same time page.evaluate(() => document.querySelector(...)) works:

const puppeteer = require("puppeteer");

(async () => {
  const browser = await puppeteer.launch();
  const [page] = await browser.pages();

  for (let i = 0; i < 200; i++) {
    await page.goto("http://www.example.com/?page=" + i);
    console.count("i");
    try {
      await page.waitForSelector("body");
    } catch (error) {
      const exists = await page.evaluate(() =>
        Boolean(document.querySelector("body"))
      );
      console.log(error.name);
      console.log("exists: " + exists);
    }

    await page.waitFor(500);
  }

  await browser.close();
})();

Seems like a frame._secondaryWorld (which is used in page.content, page.waitForSelector and so on) stops to receive execution context at some point. I logged Runtime.executionContextCreated event here which normally happens two times per navigation, one for default and one for isolated context. But here after number of reloads event starts to happen just once (for default context).

logs

i: 94
executionContextCreated: frameId 8848BB97D31B30DE5B85C03C971953D6, type default
executionContextCreated: frameId 8848BB97D31B30DE5B85C03C971953D6, type isolated
i: 95
executionContextCreated: frameId 8848BB97D31B30DE5B85C03C971953D6, type default
executionContextCreated: frameId 8848BB97D31B30DE5B85C03C971953D6, type isolated
i: 96
executionContextCreated: frameId 8848BB97D31B30DE5B85C03C971953D6, type default
executionContextCreated: frameId 8848BB97D31B30DE5B85C03C971953D6, type isolated
i: 97
executionContextCreated: frameId 8848BB97D31B30DE5B85C03C971953D6, type default
executionContextCreated: frameId 8848BB97D31B30DE5B85C03C971953D6, type isolated
i: 98
executionContextCreated: frameId 8848BB97D31B30DE5B85C03C971953D6, type default
executionContextCreated: frameId 8848BB97D31B30DE5B85C03C971953D6, type isolated
i: 99
executionContextCreated: frameId 8848BB97D31B30DE5B85C03C971953D6, type default
executionContextCreated: frameId 8848BB97D31B30DE5B85C03C971953D6, type isolated
i: 100
executionContextCreated: frameId 8848BB97D31B30DE5B85C03C971953D6, type default
i: 101
TimeoutError
exists: true
executionContextCreated: frameId 8848BB97D31B30DE5B85C03C971953D6, type default
i: 102
TimeoutError
exists: true
executionContextCreated: frameId 8848BB97D31B30DE5B85C03C971953D6, type default
i: 103
TimeoutError
exists: true
executionContextCreated: frameId 8848BB97D31B30DE5B85C03C971953D6, type default
i: 104
TimeoutError
exists: true

[vladzen13]

puppeteer@1.12.2
same problem. page.content() freezes, some waitForSelector() fall by timeout.
happens from time to time.
going down to 1.11.0

[JohnDotOwl]

puppeteer@1.12.2
same problem. page.content() freezes, some waitForSelector() fall by timeout.
happens from time to time.
going down to 1.11.0

Did 1.11.0 help solve the issue?

[vladzen13]

puppeteer@1.12.2
same problem. page.content() freezes, some waitForSelector() fall by timeout.
happens from time to time.
going down to 1.11.0

Did 1.11.0 help solve the issue?

yeah, 1.11.0 works perfectly!

[nickliqian]

puppeteer@1.7.0 has no freeze issue, too.

[aslushnikov]

Upstream fix has landed as https://chromium-review.googlesource.com/c/chromium/src/+/1496759; this'll be fixed after a roll.

[n1xn]

Can confirm - downgrading to 1.11.0 solves freezing. (in my case caused by waitForSelector() in v 1.13.0 with Chrome )

[weilinzung]

downgrading to 1.11.0 solves freezing

[peteypivots]

So iv'e been experiencing the same issue for some time. Tried many different approaches but lately have been just changing document/top location property to switch pages. Just need to be careful if the page hasn't loaded yet if your queuing up another url coming through the pipe.

await mGlobalPage.evaluate(function(url){
top.location = url;
},your_url);

[eric-zeng]

I'm having a similar issue - I'm using frame.content() to get the HTML content of iframes, and that call usually hangs. I reverted to 1.11.0 and it seems to work again. Unfortunately I can't provide good reproduction steps, because I encounter the problems when running on pages with 3rd party ads in iframes.

[aslushnikov]

@eric-zeng this should be working just fine on Puppeteer v1.14.0

[eric-zeng]

Updating to 1.14.0 solved my iframe issue, thanks! Somehow didn't notice the new release.

[royzhang666]

@aslushnikov, I also encountered the same issue with v1.13.0 and v1.14.0 that waitforselector only works for 100 times. reverted to v1.11.0 worked as expected.

[markusrempfer]

I also get a timeout error on a ... await page.waitForSelector('div#contentDiv'); ... when looping over some sub pages. I recently upgraded to v.1.14.0 ... I think from v.1.11.0 where I did not have this issue. I went back to v.1.11.0 and it's working again.

[pendo324]

This still happens to me on HeadlessChrome/79.0.3945.79 with puppeteer-core 2.0.0.

If I replace my page.waitFors with

await page.evaluate(() =>
  Boolean(document.querySelector('...'))
);

my code works as expected.

Downgrading to 1.11.0 also works as expected.

[Mahanchello]

@pendo324 @vsemozhetbyt i'm trying to figure out how to pass variable to querySelector. I'm referencing to selector the following way: sel.sub.confirmBtn. tried

await page.evaluate(() =>
  Boolean(document.querySelector(`${sel.sub.confirmBtn}`))
);

no success.
will be helpful to get answer on that 🙏

[vsemozhetbyt]

@Mahanchello Try this:

await page.evaluate(selector =>
  Boolean(document.querySelector(selector)),
  sel.sub.confirmBtn
);

[Mahanchello]

@vsemozhetbyt helped, thanks 😄

[jnre]

@pendo324 solution on puppeteer-core v2 headless works great. page.waitforselector or iframe works periodically and result in timeout, especially for nested iframe.

[Kradirhamik]

Happening to me still with v2.0

[melmarcondes]

Yes, it's happening to me still with v2.0 too

[neerajdixit27]

@aslushnikov Does the fix mentioned above rely on some lib that could be not getting installed in Docker? I am unable to reproduce this locally but when it gets deployed into a docker container, It happens more often than not. Could this be a possible reason we are still seeing this after a fix was deployed?

@djErock : I am also facing the similar issue, it works fine for me in local machine, but when we run it as docker image i see timeout issues for page.waitforselector. Can you please help, if you were able to find some solution for it. Thanks

[talkl]

Thanks guys! After looking threw some 150-200 github issues, posts (stackoverflow and otherwise). This was the one that helped me. There are currently 621 issues for this repository.

By my estimates the _otherworld or isolated context doesn't get updated after a while. So you really need to query the actual browser with page.evaluate.

page.context() is also outdated after a few page visits in Docker. Anyone looking into Puppeteer should probably also looking into https://github.com/cyrus-and/chrome-remote-interface. This repository has 1 open issue a.t.m.

Although I tried really hard not to look into the source code. I assume the event loop is blocked for duration of the wait. My solution was recursive Promise-polling with page.evaluate with setTimeout leave the eventloop generally still available. And that seemed to work.

how exactly did you solve it? can you show the code?

[ShefaliChopkar]

@aslushnikov Does the fix mentioned above rely on some lib that could be not getting installed in Docker? I am unable to reproduce this locally but when it gets deployed into a docker container, It happens more often than not. Could this be a possible reason we are still seeing this after a fix was deployed?

@djErock : I am also facing the similar issue, it works fine for me in local machine, but when we run it as docker image i see timeout issues for page.waitforselector. Can you please help, if you were able to find some solution for it. Thanks

We are also facing timeouts issues on jenkins, which is running on docker image. can you help us understand what worked for you?

[jmrnilsson]

Thanks guys! After looking threw some 150-200 github issues, posts (stackoverflow and otherwise). This was the one that helped me. There are currently 621 issues for this repository.
By my estimates the _otherworld or isolated context doesn't get updated after a while. So you really need to query the actual browser with page.evaluate.
page.context() is also outdated after a few page visits in Docker. Anyone looking into Puppeteer should probably also looking into https://github.com/cyrus-and/chrome-remote-interface. This repository has 1 open issue a.t.m.

Although I tried really hard not to look into the source code. I assume the event loop is blocked for duration of the wait. My solution was recursive Promise-polling with page.evaluate with setTimeout leave the eventloop generally still available. And that seemed to work.

how exactly did you solve it? can you show the code?

I downgraded to 0.11, I believe. Then replaced with playwrite I think. These issues brought back some early memories from Splinter.

I also ensured we didn’t open tabs right and left.

[Plinpod]

I am still having this exact same issue as the original post. Running puppeteer 10.2.0? Any thoughts?

[industriousparadigm]

Hello! Same here, puppeteer worked great for several iterations of going through a page, then after N loops it will hang at waitForSelector(). Running 10.2.0 as well.

[andikawhy]

await page.content(); also hanging forever for me in nodejs. this is the environment

node: 12
puppeteer: ~5.2.1,
puppeteer-extra: ~3.1.13,
puppeteer-extra-plugin-stealth: ~2.4.15,

any thought about this? thanks!