Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Will it be delayed for 20 seconds? #11

Closed
kanxue660 opened this issue Sep 6, 2018 · 5 comments
Closed

Will it be delayed for 20 seconds? #11

kanxue660 opened this issue Sep 6, 2018 · 5 comments
Labels
bug Something isn't working

Comments

@kanxue660
Copy link

The code is as follows
const { Cluster } = require('puppeteer-cluster');
(async () => {
const cluster = await Cluster.launch({
concurrency: Cluster.CONCURRENCY_CONTEXT,
maxConcurrency: 2,
sameDomainDelay:20*1000 //Will it be delayed for 20 seconds?
});

await cluster.task(async ({ page, data: url }) => {
await page.goto(url);
const screen = await page.screenshot();
// Store screenshot, do something else
});

await cluster.queue('http://www.google.com/a.html');
await cluster.queue('http://www.google.com/b.html');
await cluster.queue('http://www.google.com/c.html');
// many more pages

await cluster.idle();
await cluster.close();
})();

My question is, if a.HTML opens first, then B.Html,c.HTML, will be delayed 20 seconds to open it?
Do not understand how this sameDomainDelay uses

@thomasdondorf thomasdondorf added the bug Something isn't working label Sep 6, 2018
@thomasdondorf
Copy link
Owner

Thanks you for asking. This is a bug. Normally it should work like this:

A -> wait for 20 sec -> B -> wait for 20 sec -> C

But currently (buggy behavior) it works like this (in case there are 2 workers):
A|B -> wait for 20 sec -> C

This is not intended. Will be fixed soon.

@thomasdondorf
Copy link
Owner

thomasdondorf commented Sep 6, 2018

Have a look at version v0.11.1 which fixes the bug. There will now be a delay 20 seconds in your example between the task function calls. Feel free to reopen if anything is not working.

@kanxue660
Copy link
Author

I updated to the latest version,
I found that when the program was waiting, the CPU utilization reached 35%, which was a bit abnormal.

     My computer configuration is:
      CPU i5-3470, 
      windows 10 x 64,
      memory: 20 gigabytes

@thomasdondorf
Copy link
Owner

Will investigate (see #13).

@thomasdondorf
Copy link
Owner

Fixed in v0.11.2 (just released). Thank you for finding all these little bugs @kanxue660 :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants