Repository navigation
SEMRUSH now hiding their SEO scanners as Fake Google Bots #328
Description
SEO company SEMRUSH is now hiding their SEO audit scanners as fake Googlebots.
Found in recent logs all posing as Googlebot
46.229.173.66 siteaudit.crawler.semrush.com.
46.229.173.67 siteaudit.crawler.semrush.com.
46.229.173.68 siteaudit.crawler.semrush.com.
Activity
VedranIteh commentedon Dec 16, 2019
fail2bans apache-fakegooglebot jail resolves this efficiently with a host resolving script:
https://github.com/fail2ban/fail2ban/blob/0.9/config/filter.d/ignorecommands/apache-fakegooglebot
mitchellkrogza commentedon Dec 17, 2019
Thanks @VedranIteh will check it out. Have doubts however about how it deals with no hostname. I create my own list of Fake Googlebots through this script - I sometimes sit and go through each and every one (yawn) and find quite a few IP's that do not resolve to a hostname but do belong to Google Inc. This filter would then block them right?
VedranIteh commentedon Dec 17, 2019
great approach, I'm gonna try it on my logs and report back.
VedranIteh commentedon Dec 17, 2019
Hey Mitchell, checked the hostless (and other) IP's banned by apache-fakegooglebot script, grep'd the abuse emails to see who they belong to:
fail2ban-client status apache-fakegooglebot | egrep -o "([0-9]{1,3}\.){3}[0-9]{1,3}" | xargs -n 1 whois | grep abuse-mailboxand got 0 real google bots, but my sample might be way to small for a definite conclusion. Your script (findfakebots) seem to be simillar concept as the fail2bans one, so my suggestion was pointless from the beginning 🥇 .
One last thing, google seems to suggest the bots shouldn't be hostless and as an addition to your script the suggestion is to include forward resolving after getting the host
https://support.google.com/webmasters/answer/80553?hl=en
Great stuff on your repos ! gonna keep lurking...