Timeout config not honored

[bangbambang]

Related code:

  const cluster = await Cluster.launch({
    puppeteerOptions: {
      headless: true,
      ignoreHTTPSErrors: env.IGNORE_HTTPS || false,
      args: ['--disable-http2'],
      timeout: env.PUPPETEER_TIMEOUT || 60000,            //attempt 2
    },
    concurrency: Cluster.CONCURRENCY_CONTEXT,
    maxConcurrency: parseInt(env.MAX_WORKER) || 4,
    skipDuplicateUrls: false,
    monitor: env.MONITOR === 'true' || false,
    timeout: env.PUPPETEER_TIMEOUT || 60000,              //attempt 1
  });

I've tried to set timeout in cluster launch options and passed it to puppeteerOptions, all failed. Log says timeout was still at 30000.

app:cluster:err TimeoutError: Navigation Timeout Exceeded: 30000ms exceeded
  app:cluster:err     at Promise.then (/home/bambang/project/om-screenshoot/node_modules/puppeteer/lib/FrameManager.js:1276:21)
  app:cluster:err   -- ASYNC --
  app:cluster:err     at Frame.<anonymous> (/home/bambang/project/om-screenshoot/node_modules/puppeteer/lib/helper.js:144:27)
  app:cluster:err     at Page.goto (/home/bambang/project/om-screenshoot/node_modules/puppeteer/lib/Page.js:624:49)
  app:cluster:err     at Page.<anonymous> (/home/bambang/project/om-screenshoot/node_modules/puppeteer/lib/helper.js:145:23)
  app:cluster:err     at GenericHandler.processPage (/home/bambang/project/om-screenshoot/src/handlers/v1base.js:47:21)
  app:cluster:err     at GenericHandler.process (/home/bambang/project/om-screenshoot/src/handlers/v1base.js:94:16)
  app:cluster:err     at module.exports (/home/bambang/project/om-screenshoot/src/handlers/site.js:27:24)
  app:cluster:err     at Worker.<anonymous> (/home/bambang/project/om-screenshoot/node_modules/puppeteer-cluster/dist/Worker.js:56:54)
  app:cluster:err     at Generator.next (<anonymous>)
  app:cluster:err     at fulfilled (/home/bambang/project/om-screenshoot/node_modules/puppeteer-cluster/dist/Worker.js:4:58)
  app:cluster:err     at process.internalTickCallback (internal/process/next_tick.js:77:7) +789ms

Any guidance on how to trace/ fix this issue?

[thomasdondorf]

The timeout options specified in the launch function is the timeout for the whole task not for individual goto calls. If you want to specify the timeout for a specific goto, checkout the puppeteer docs.

[bangbambang]

Ah, I see.

In that case, is it correct to assume that I'll have to set the timeout on Cluster.launch to be at least as long as the goto timeout to prevent early termination due to cluster timeout being less than navigation timeout?

[thomasdondorf]

Yes, exactly. The timeout option of the cluster is mostly useful if you have more complex tasks (in case you want to navigate multiple times or do other asynchronous actions inside of the task).

[bangbambang]

OK. Thanks for the info, much appreciated.