Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Safe per IP Google limits? #19

Open
sf-steve opened this issue Apr 22, 2019 · 3 comments
Open

Safe per IP Google limits? #19

sf-steve opened this issue Apr 22, 2019 · 3 comments
Labels
good first issue Good for newcomers question Further information is requested

Comments

@sf-steve
Copy link

Hi all, does anyone have up-to-date data for how many searches you can perform per IP per time period before getting blocked?

We have a limited scraping need, and spare server resources, so figured this would be a good solution, but would like to know how best to split the work.

I found old posts suggesting around 300 regular (not Google dorks) searches per 24 hours, but have no idea if this is still correct.

Any input greatly appreciated.

@YvesBos
Copy link

YvesBos commented Apr 23, 2019

I have been able to scrape for +12 hours with a 70-200 second sleep in between each query on a single IP, without detection. When I reduced the sleep range to 60-110 seconds, it was detected after ~15 minutes. Patience is key :)

@sf-steve
Copy link
Author

Thanks @YvesBos, so thats inline with the older blog posts i found. I guess we can set up a couple of VPS's and run some tests.
Have you tried any tests with continuous sessions (same userdatadir) vs fresh sessions?

@YvesBos
Copy link

YvesBos commented Apr 24, 2019

Ah yes, I forgot to mention that. I only scrape for a couple of keywords and then relaunch the scraper with a different country in the config, thus creating a new session. I have no experience with using continuous sessions.

@NikolaiT NikolaiT added good first issue Good for newcomers question Further information is requested labels May 6, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
good first issue Good for newcomers question Further information is requested
Projects
None yet
Development

No branches or pull requests

3 participants