Should we increase default timeout?

[rgaudin]

Scraping fondamentaux, I got a lot of timeouts on pages (169!) with the default 30s timeout.

@ikreymer could you please give us more details on the implication of timeout:

• What does it apply to? await page.goto(url, {waitUntil, timeout}); implies it's the time for the page to load (that's the default waitUntil value). Is this the time before the document loaded event?
• What we be the consequences of increasing it?
• How are non-page requests affected? Like video files for example.

My initial though would be to use a large value here to capture long pages and other pages would still load and be processed fast.

[ikreymer]

• What does it apply to? await page.goto(url, {waitUntil, timeout}); implies it's the time for the page to load (that's the default waitUntil value). Is this the time before the document loaded event?

Yes, the timeout applies to the various 'modes' of the goto function.
It could wait for either just the dom loading, or until there is no more network traffic.
The options are better explained here:
https://github.com/puppeteer/puppeteer/blob/v5.3.1/docs/api.md#pagegotourl-options

• What we be the consequences of increasing it?

It depends. Maybe its good to just set it to 1 min. For sites that load quickly, there should be no affect.

• How are non-page requests affected? Like video files for example.

They would not be included in this.. There will be separate logic, part of #9, to handle waiting for video playing.

Also, the timeouts could be for any number of reasons.. for example, I've seen more timeouts when too many browsers are running and there is not enough CPU..

[rgaudin]

OK, thanks for the details. I think we should increase the default then. Maybe 90s?

I think we shall also document in the --scroll option help its custom timeout.

You mentioned the separate behavior for video but what about non-video resources? Images, or large non-media files like JS or WASM files? Is there a timeout at play there?

Thanks for pointing to the concurrency issue ; indeed that must be taken into consideration when writing recipes. Network is also affected although I understand for most cases, bottleneck would be CPU.