Resolv issues with googlebot sometimes

[inspire22]

Thanks again for this project :)

I've been getting this sometimes now:
DNS result has no information for crawl-95-216-33-117.googlebot.com"

I can rescue nil inside the rack_attack Legitbot.bot call, but would love to solve the actual problem as well.

It's strange that it says "no information" but then clearly has resolved it to crawl-95-216-33-117.googlebot.com

Hmm, maybe reverse-dns is working to get the address, but then it's not able to ping it?

The IP reported in my error logs is in fact 95.216.33.117

/usr/local/rvm/rubies/ruby-2.7.2/lib/ruby/2.7.0/resolv.rb:379:in `getaddress'
/u/apps/ap.next/shared/bundle/ruby/2.7.0/gems/legitbot-1.5.1/lib/legitbot/validators/domains.rb:66:in `reverse_ip'
/u/apps/ap.next/shared/bundle/ruby/2.7.0/gems/legitbot-1.5.1/lib/legitbot/validators/domains.rb:48:in `valid_domain?'
/u/apps/ap.next/shared/bundle/ruby/2.7.0/gems/legitbot-1.5.1/lib/legitbot/validators/domains.rb:22:in `valid_domain?'
/u/apps/ap.next/shared/bundle/ruby/2.7.0/gems/legitbot-1.5.1/lib/legitbot/botmatch.rb:25:in `valid?'
/u/apps/ap.next/shared/bundle/ruby/2.7.0/gems/legitbot-1.5.1/lib/legitbot/botmatch.rb:29:in `fake?'
/u/apps/ap.next/current/config/initializers/rack_attack.rb:16:in `block in '
/u/apps/ap.next/shared/bundle/ruby/2.7.0/gems/rack-attack-6.5.0/lib/rack/attack/check.rb:15:in `matched_by?'
/u/apps/ap.next/shared/bundle/ruby/2.7.0/gems/rack-attack-6.5.0/lib/rack/attack/configuration.rb:72:in `block in blocklisted?'
/u/apps/ap.next/shared/bundle/ruby/2.7.0/gems/rack-attack-6.5.0/lib/rack/attack/configuration.rb:72:in `any?'
/u/apps/ap.next/shared/bundle/ruby/2.7.0/gems/rack-attack-6.5.0/lib/rack/attack/configuration.rb:72:in `blocklisted?'
/u/apps/ap.next/shared/bundle/ruby/2.7.0/gems/rack-attack-6.5.0/lib/rack/attack.rb:107:in `call'
/u/apps/ap.next/shared/bundle/ruby/2.7.0/gems/newrelic_rpm-8.3.0/lib/new_relic/agent/instrumentation/middleware_tracing.rb:100:in `call'
/u/apps/ap.next/shared/bundle/ruby/2.7.0/gems/rack-2.2.3/lib/rack/tempfile_reaper.rb:15:in `call'
/u/apps/ap.next/shared/bundle/ruby/2.7.0/gems/newrelic_rpm-8.3.0/lib/new_relic/agent/instrumentation/middleware_tracing.rb:100:in `call'

[alaz]

Hi,

My guess is that is is an inconsistency between the forward and the reverse DNS records. Most likely Google dynamically and frequently updates the crawler's DNS records. So my theory is that sometimes the forward resolves correctly, but the reverse returns NXDOMAIN because it was not in the cache and hence the result is more up-to-date.

Does it make sense to you?

[inspire22]

Interesting. I have only a loose understanding of dns but I'd thought it'd propagate up to a master copy if it can't find a record?

Any tips how to solve it best in a rack-attack block? Or should it be fine to just rescue the exception?

[alaz]

I'd thought it'd propagate up to a master copy if it can't find a record?

Yes. Local / closest DNS servers cache records. Consider your closest DNS server has a forward record in its cache and returns it. But it has to go further in the chain to get the result for a reverse query and gets NXDOMAIN in return, because it does not exist anymore.

[inspire22]

Does rescuing the exception seem like a decent plan, or does it open up other bots? Or should this be something rescued in the gem itself.

[alaz]

Yes, rescuing the exception in a calling code might open the door for other bots. It should be rescued in the same way as for forward resolv:

legitbot/lib/legitbot/validators/domains.rb

Lines 51 to 55 in 4a98be8

	def reverse_domains(ip)
	resolver.getnames(ip)
	rescue Resolv::ResolvError
	nil
	end

[alaz]

Published as 1.6.0

[inspire22]

Awesome, thanks!