-
Notifications
You must be signed in to change notification settings - Fork 10
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add missing Google crawlers #85
Comments
I'm getting googlebot blocked quite a bit in my rack-attack logs using legitbot, it's probably because some IPs are missing? Is it possible to automate the process of adding new IPs using the host command like they suggest here? I actually had more IPs in there but calling 'host' on them I realized they were actually fake & I'd just randomly had the first ones I tested be actually googlebot. |
@inspire22 Legitbot follows the exact verification procedure you linked to, only programmatically. Did you try to follow the steps? These IPs do not pass for me:
|
Oops, you're right, thanks! Strange they would match the first step and not the second. I'd mistaken your TODO to add crawlers for adding crawler IPs, which is why I jumped on here. My bad and apologies :) |
By the way, I don't think these IPs belong to Google. Both of them are owned by Hetzner (a well known European hosting provider):
Someone managed to convince Hetzner to create these reverse DNS records (I am surprised). Faking corresponding forward records is close to impossible, as Google itself controls the zone. |
List of the crawlers
The text was updated successfully, but these errors were encountered: