New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Jobs to poll #173
Comments
I'd expect developers in this case to write spiders |
Actually it makes sense for the poller |
By the way, you can set the poller to poll on a sub-second interval. |
Today we can choose "poll_interval", "max_proc" and "max_proc_per_cpu". "max_proc" and "max_proc_per_cpu" are for limiting a maximum of jobs that can be running.
If "poll_interval" is high that maximum would never be reached. That happens because after each poll interval we start only one job.
E.g.: poll_interval = 30 and some spider takes about 2 minutes to finish. The maxium jobs that would run in this case is 4~5, no matter what "max_proc" and "max_proc_per_cpu" values are.
Maybe it should have an option "jobs_to_poll". With that we could choose how many jobs to poll each time, instead of only one.
What you guys think about that?
Thank you
The text was updated successfully, but these errors were encountered: