Timeout exception despite all requests being done and using networkidle0 option

[ZigaStrgar]

Steps to reproduce

Tell us about your environment:

• Puppeteer version: 1.10.0
• HeadlessChrome: 72.0.3582.0
• Platform / OS version: Ubuntu running in docker container
• Node.js version: 8.11.4
• Yarn version: 1.6.0

What steps will reproduce the problem?

const browser = await puppeteer.launch({args: ['--no-sandbox', '--disable-setuid-sandbox']});
const page = await browser.newPage();

await page.authenticate({
    username: process.env.BASIC_AUTH_USERNAME,
    password: process.env.BASIC_AUTH_PASSWORD
});

const q = JSON.stringify(query)

await page.goto(`${process.env.APP_URL}/screenshot?graph=${chart}&type=${type}&q=${q}`, {
     waitUntil: 'networkidle0',
});

await page.screenshot({path: `./storage/app/public/screenshots/${path}.png`});

await browser.close();

What is the expected result?
I'm trying to create a screenshot of a chart build from the URL parameters. And save it to the drive and later include it in the automated email reporting service. At the requested webpage the only 3 items loading are favicon.ico, app.js as the main JS file containing all the Vue components and libraries used in the process and the last one is an AJAX request to the backend where it gets and manipulates the data to correct format for a given chart.

While visiting the URL in browser get's me the "final" product, the script fails.

What happens instead?
I receive the TimeoutError but when profiling the page all the requests finish in less than 30 seconds as that is the default setting.

screenshot_1  |  { TimeoutError: Navigation Timeout Exceeded: 30000ms exceeded
screenshot_1  |  at Promise.then (/var/www/node_modules/puppeteer/lib/FrameManager.js:1230:21)
screenshot_1  |  at <anonymous>
screenshot_1  |  -- ASYNC --
screenshot_1  |  at Frame.<anonymous> (/var/www/node_modules/puppeteer/lib/helper.js:144:27)
screenshot_1  |  at Page.goto (/var/www/node_modules/puppeteer/lib/Page.js:587:49)
screenshot_1  |  at Page.<anonymous> (/var/www/node_modules/puppeteer/lib/helper.js:145:23)
screenshot_1  |  at /var/www/screenshot.js:18:24
screenshot_1  |  at <anonymous>
screenshot_1  |  at process._tickCallback (internal/process/next_tick.js:188:7) name: 'TimeoutError' }

Please take note that I did NOT include the code of whole file screenshot.js so that 18:24 may be wrong but it is pointing to the line await page.goto

Further notes to take into consideration

1. I've developed the code on the following environment and everything works:
   • Puppeteer version: 1.10.0
   • HeadlessChrome: 72.0.3582.0
   • Platform / OS version: Mac OSX Mojave
   • Node.js version: 11.1.0
   • Yarn version: 1.12.3
2. I've rerolled the production version back to last working commit and gradually add the code of latest version and my code went through as it should but after running yarn upgrade which upgraded HeadlessChrome from 71.0.3563.0 to 72.0.3582.0 the problem arrived
3. I've added the code for InflightRequestInspection from Navigation Timeout Exceeded when using networkidle0 and no insight into what timed out #1908 more exactly Navigation Timeout Exceeded when using networkidle0 and no insight into what timed out #1908 (comment) and the array of requests printed in the catch was empty so all requests were done. I've tested this with 300ms timeout limit and it printed out the AJAX request URL.
4. I'm not sure if the issue is with puppeteer or HeadlessChrome or just something else. But I'm first posting the issue here.
5. The last version of puppeteer used in the production environment while everything worked was 1.9.0

---

Update (28th Novembre)

Again rerolled the version back to working and only upgraded the puppeteer package with yarn upgrade puppeteer and the issue persisted. This is the yarn log that was outputted while doing that.

yarn upgrade v1.12.3
[1/4] 🔍  Resolving packages...
[2/4] 🚚  Fetching packages...
[3/4] 🔗  Linking dependencies...
warning "laravel-mix > img-loader@3.0.1" has unmet peer dependency "imagemin@^5.0.0 || ^6.0.0".
[4/4] 📃  Rebuilding all packages...
success Saved lockfile.
success Saved 20 new dependencies.
info Direct dependencies
└─ puppeteer@1.10.0
info All dependencies
├─ async-limiter@1.0.0
├─ brace-expansion@1.1.11
├─ buffer-from@1.1.1
├─ concat-map@0.0.1
├─ es6-promise@4.2.5
├─ es6-promisify@5.0.0
├─ extract-zip@1.6.7
├─ fd-slicer@1.0.1
├─ fs.realpath@1.0.0
├─ inflight@1.0.6
├─ mime@2.4.0
├─ pend@1.2.0
├─ process-nextick-args@2.0.0
├─ progress@2.0.1
├─ proxy-from-env@1.0.0
├─ puppeteer@1.10.0
├─ typedarray@0.0.6
├─ util-deprecate@1.0.2
├─ ws@5.2.2
└─ yauzl@2.4.1

And the only package that changed alongside with puppeteer was mime which was upgraded from 2.3.1 to 2.4.0

[aslushnikov]

@ZigaStrgar so I read this as if page.goto with networkidle0 option doesn't work for certain URL. Can you share the URL so that I can see what's going on?

[ZigaStrgar]

@aslushnikov I'm sorry but I can't share the specific URL with you since it's an internal application. But I can tell you that on the page is rendered only a canvas element and it's being manipulated by Chart.js library.

I've also opened an issue on chromium.org also and there is a guy saying that networkidle0 doesn't fire when page is using WebGL.

---

Raw HTML of the page:

<html>
  <head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, user-scalable=no, initial-scale=1.0, maximum-scale=1.0, minimum-scale=1.0">
    <meta http-equiv="X-UA-Compatible" content="ie=edge">
    <title>Screenshot</title>
    <meta name="csrf-token" content="{{ csrf_token() }}">
  </head>
  <body>
    <div id="root">
      <url-graph url="/custom/graphs/data/{{ $type }}"></url-graph>
    </div>
    <script src="/js/app.js"></script>
  </body>
</html>

Template of <url-graph> Vue component

<template>
    <div>
        <canvas width="800" height="600" ref="graph"></canvas>
    </div>
</template>

[aslushnikov]

@ZigaStrgar ah, I see you use page.authenticate, that uses request interception internally. Can you try running your script without page.authenticate and see if it works? This looks very much like #3471.

I've also opened an issue on chromium.org also and there is a guy saying that networkidle0 doesn't fire when page is using WebGL.

I don't see any reasons why this might happen.

[aslushnikov]

Closing this as a dupe of #3471.