robots.txt file inspection
Switch branches/tags
Nothing to show
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
lib
spec
.rspec
.ruby-gemset
.ruby-version
.travis.yml
Gemfile
Gemfile.lock
README.md
Rakefile
irobot.gemspec

README.md

irobot

Team Shared Services Status Production Open Source Critical True

A robots.txt file inspector.

Configuration

An example of some common config options include:

require 'simple_redis_cache'

Irobot.configure do |c|
  c.timeout = 1 # second
  c.cache_namespace
  c.cache = SimpleRedisCache.new(ttl: 1.day)
  c.logger = Logger.new(File.join(ROOT, 'log', 'irobot.log'))

  # Since we aren't actually crawling the site, we'll ignore crawl delays
  c.respect_crawl_delay = false
end

Usage

--- moz/irobot » bundle exec pry
[1] pry(main)> require 'irobot';
[2] pry(main)> Irobot.allowed?('http://amazon.com', 'EtaoSpider')
=> false
[3] pry(main)> Irobot.allowed?('http://moz.com', 'mozbot')
=> true