Navigation Timeout Exceeded when using networkidle0 and no insight into what timed out

[peterbe]

Steps to reproduce

Tell us about your environment:

• Puppeteer version: 1.0.0
• Platform / OS version: macOS 10.13.2
• URLs (if applicable): Would be nice to know *what* timed out peterbe/minimalcss#112 https://gist.githubusercontent.com/peterbe/c7c21131fbe23031034fea114dfd6b97/raw/1f0a8329bb74127f9cb286f727f4b3246a43b70e/default.txt https://songsear.ch/
• Node.js version: 8.3.0

What steps will reproduce the problem?

'use strict'
const puppeteer = require('puppeteer')
;(async () => {
  const browser = await puppeteer.launch({})
  const page = await browser.newPage()
  await page.setRequestInterception(true)
  page.on('request', request => request.continue())

  try {
    await page.goto('https://songsear.ch/', {
      waitUntil: 'networkidle0',    // fails
      // waitUntil: 'networkidle2',    // works
      timeout: 2000
    })
  } catch (ex) {
    await browser.close()
    console.warn('IT FAILED')
    console.error(ex.toString())
    throw ex
  }
  await browser.close()
})()

RUnning this:

▶ node --trace-warnings hack.js
IT FAILED
Error: Navigation Timeout Exceeded: 2000ms exceeded
(node:58887) Error: Navigation Timeout Exceeded: 2000ms exceeded
    at Promise.then (/Users/peterbe/dev/JAVASCRIPT/minimalcss/node_modules/puppeteer/lib/NavigatorWatcher.js:71:21)
    at <anonymous>
(node:58887) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
    at emitWarning (internal/process/promises.js:69:15)
    at emitPendingUnhandledRejections (internal/process/promises.js:86:11)
    at runMicrotasksCallback (internal/process/next_tick.js:124:9)
    at _combinedTickCallback (internal/process/next_tick.js:131:7)
    at process._tickCallback (internal/process/next_tick.js:180:9)

This fails every time. It doesn't matter if I change 2000 to 20000 or 30000.

If you change from networkidle0 to networkidle2 it works every time.
But that's not a "solution" because I like networkidle0 since I want to wait for all its resources to be downloaded so they can be analyzed and stuff. That "requirement" is not included in the script above.

The origin of this problem is here: peterbe/minimalcss#112

Running that failing script with full debugging turns up this:
https://gist.githubusercontent.com/peterbe/c7c21131fbe23031034fea114dfd6b97/raw/1f0a8329bb74127f9cb286f727f4b3246a43b70e/default.txt

What is the expected result?
Some insight into what resource got stuck when it gives up waiting for something after 500ms.

What happens instead?
Failure without insight into what resource is being problematic.

PS. I own https://songsear.ch and it works in all browsers that I know of but if it helps the cause can I fiddle with that site, to some degree, to figure this out for the sake of puppeteer.

[dan335]

I'm getting this Navigation Timeout Exceeded error seemingly randomly. Even using networkidle2. Using 1.0 in docker.

[tomgallagher]

My limited experience of Navigation Timeout Exceeded errors is that you can get them for a huge number of reasons. Some sites seem to have multiple open connections that even networkidle2 won't allow for. If you go for the load event and test the same sites, this will tell you whether any given site has this characteristic. Also, timeout exceeded error will occur seemingly randomly where you are operating at the limit of your system resource limitations, whether that be in terms of your memory usage or network connections.

[peterbe]

This isn't about preventing it from happening. It might happen because of a bug in puppeteer or chromium. Or, it could be because of a a bug in your site that you try to open because it as a resource reference to a VPN only IP or something.

What this issue is about is; What URL (or thing) was it that took so long to wait for that the error was raised?

The exception string is Error: Navigation Timeout Exceeded: ${config.timeout}ms exceeded. What it should be is Error: Navigation Timeout Exceeded: ${config.timeout}ms exceeded trying to download ${url}.

[tomgallagher]

OK. The more information the better, I'm sure, but my point was that it could be many things, whether internal to Chrome, external links or system resources that caused the timeout. So what would the timeout error report? The URL or URLs that happened to be processing when Chrome ran short of memory or network connections? Is that good information? I suppose an array of the remaining open network connections would at least enable the developer to eliminate certain possibilities but it could also lead to a lot of wild goose chases when none of those URLs are the cause.

[peterbe]

an array of the remaining open network connections

❤️ !

[jacksontong]

I have exactly the same problem, the thing is I don't have this problem if I run puppeteer on Mac, but when I switch to docker (linux) then I have this issue. Both Mac and docker are using the same chrome version HeadlessChrome/66.0.3347.0

[aslushnikov]

I suppose an array of the remaining open network connections would at least enable the developer to eliminate certain possibilities but it could also lead to a lot of wild goose chases when none of those URLs are the cause.

You can track this with the request, requestfailed and requestfinished events.

class InflightRequests {
  constructor(page) {
    this._page = page;
    this._requests = new Set();
    this._onStarted = this._onStarted.bind(this);
    this._onFinished = this._onFinished.bind(this);
    this._page.on('request', this._onStarted);
    this._page.on('requestfinished', this._onFinished);
    this._page.on('requestfailed', this._onFinished);
  }

  _onStarted(request) { this._requests.add(request); }
  _onFinished(request) { this._requests.delete(request); }
 
  inflightRequests() { return Array.from(this._requests); }  

  dispose() {
    this._page.removeListener('request', this._onStarted);
    this._page.removeListener('requestfinished', this._onFinished);
    this._page.removeListener('requestfailed', this._onFinished);
  }
}

and use it later like this:

const tracker = new InflightRequests(page);
await page.goto(url).catch(e => {
  console.log('Navigation failed: ' + e.message);
  const inflight = tracker.inflightRequests();
  console.log(inflight.map(request => '  ' + request.url()).join('\n'));
});
tracker.dispose();

Hope this helps.

[darron1217]

I couldn't catch any request...
Can anybody tell me why it is not working?

class InflightRequests {
  constructor(page) {
    this._page = page;
    this._requests = new Set();
    this._fails = new Set();
    this._onStarted = this._onStarted.bind(this);
    this._onFinished = this._onFinished.bind(this);
    this._onFailed = this._onFailed.bind(this);
    this._page.on('request', this._onStarted);
    this._page.on('requestfinished', this._onFinished);
    this._page.on('requestfailed', this._onFailed);
  }

  _onStarted(request) { this._requests.add(request); }
  _onFinished(request) { this._requests.delete(request); }
  _onFailed(request) { this._fails.add(request); }

  inflightRequests() { return Array.from(this._requests); }
  failedRequests() { return Array.from(this._fails); }

  dispose() {
    this._page.removeListener('request', this._onStarted);
    this._page.removeListener('requestfinished', this._onFinished);
    this._page.removeListener('requestfailed', this._onFailed);
  }
}

        const tracker = new InflightRequests(page);

        await page.goto(request.url, requestOptions).catch(e => {
          const inflight = tracker.inflightRequests();
          const fail = tracker.failedRequests();
          console.log('Remaining Requests:\n' + inflight.map(req => req.url()).join('\n'));
          console.log('Failed Requests:\n' + fail.map(req => req.url()).join('\n'));
          throw new Error(e.message);
        });
        tracker.dispose();

and error body...

The command "PATH=$PATH:/usr/local/bin NODE_PATH='/var/www/html/node_modules' node '/var/www/html/ven
  dor/spatie/browsershot/src/../bin/browser.js' '{"url":"file:\/\/\/tmp\/0695567001526042352\/index.htm
  l","action":"evaluate","options":{"pageFunction":"Object.keys(window.results).length","args":["--no-s
  andbox","--lang=ko-KR","--disable-web-security","--user-data-dir=\/chrome"],"viewport":{"width":800,"
  height":600},"displayHeaderFooter":false,"executablePath":"\/usr\/bin\/chromium-browser","waitUntil":
  "networkidle0","timeout":30000}}'" failed.

  Exit Code: 1(General error)

  Working directory: /var/www/html

  Output:
  ================
  Remaining Requests:

  Failed Requests:



  Error Output:
  ================
  (node:5768) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 1): Error: P
  age crashed!
  (node:5768) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future,
   promise rejections that are not handled will terminate the Node.js process with a non-zero exit code
  .
  Error: Navigation Timeout Exceeded: 30000ms exceeded
      at page.goto.catch.e (/var/www/html/vendor/spatie/browsershot/bin/browser.js:102:17)
      at <anonymous>

[peterbe]

I request we re-open this.

You can track this with the request, requestfailed and requestfinished events.

said @aslushnikov

In peterbe/minimalcss#199 we tried implementing a tracker so that we can extend the Navigation Timeout Exceeded: 30000ms exceeded error message with a list of URLs that got started but never finished.
Even with that, and with upgrading to puppeteer 1.4.0, it still fails with the tracker not having any unfinished URLs.

I also tried logging every single possible event (that has to do with requests) on the Page object. And that too didn't help. Still no insight into what request is exceeding the timeout.

My hunch is that it's related to service workers. The service-worker set up the site I'm testing against is "stock create-react-app" style. Nothing fancy.
I can try the same URLs by changing it to http://songsearch.local which is my local Nginx. When I do that, it always works. And since it's not HTTPS the service-worker will be automatically disabled.

So, are there tools and techniques to track (and even disable) service workers?

[konstantinblaesi]

@darron1217 make sure you handle this event https://github.com/GoogleChrome/puppeteer/blob/v1.6.0/docs/api.md#event-error
I'd handle this one too https://github.com/GoogleChrome/puppeteer/blob/v1.6.0/docs/api.md#event-disconnected
Unhandled 'error' events will cause rejected promises, hence the "UnhandledPromiseRejectionWarning" in the log.

[darron1217]

@konstantinblaesi Yes, I solved it by handling error in right way :)
Thanks

[diachedelic]

Thanks @aslushnikov, in my case Webpack's Hot Module Reload remained connected: http://localhost:8009/__webpack_hmr

[aslushnikov]

Even with that, and with upgrading to puppeteer 1.4.0, it still fails with the tracker not having any unfinished URLs.

@peterbe that's due to #3471 . Please upvote to follow updates.

[jacobweber]

Ran into the same problem, also with create-react-app. I used the above tracker code to find out that the sockjs-node calls were keeping networkidle0 from firing.

This happened on CentOS 7, although it didn't happen on Mac OS 10.14.1.

Is there any way to exclude those calls from being considered? Switching to waitUntil=load worked, but that's not ideal, since it's a single-page app.

[diachedelic]

@jacobweber something like this might help, unless your sockjs-node calls are required for your page to load correctly:

      await page.setRequestInterception(true)

      page.on('request', req => {
        // disable webpack HMR, which breaks the 'networkidle0' setting
        if (req.url().endsWith('/__webpack_hmr')) {
          req.abort()
        } else {
          req.continue()
        }
      })

[ColCh]

@diachedelic #1908 (comment) OMG lifesaver! Thank you, that was totally unobvious!

Nothing to do with puppeteer here

[zhangzippo]

I have exactly the same problem, the thing is I don't have this problem if I run puppeteer on Mac, but when I switch to docker (linux) then I have this issue. Both Mac and docker are using the same chrome version HeadlessChrome/66.0.3347.0

I also has the problem. on mac is run success when to centos it's fail, Did you solve it ?

[yes1am]

I have exactly the same problem, the thing is I don't have this problem if I run puppeteer on Mac, but when I switch to docker (linux) then I have this issue. Both Mac and docker are using the same chrome version HeadlessChrome/66.0.3347.0

I also has the problem. on mac is run success when to centos it's fail, Did you solve it ?

me too.

[davidtorroija]

@jacobweber something like this might help, unless your sockjs-node calls are required for your page to load correctly:

      await page.setRequestInterception(true)

      page.on('request', req => {
        // disable webpack HMR, which breaks the 'networkidle0' setting
        if (req.url().endsWith('/__webpack_hmr')) {
          req.abort()
        } else {
          req.continue()
        }
      })

Awesome thanks for this solution!

[afkoziol]

I was experiencing this timeout as well.

In my case, I was intercepting requests and adding a custom header. For some reason, GET requests of some font files got hung and the timeout popped. This would only happen in a docker env (chromium-browser in Alpine) and only on certain pages (same webpack dev-server instance running).

I changed the header add to only be included if the request was 'x-requested-with' XMLHttpRequest. And the timeout went away. not sure if this helps anyone but maybe?