Ruby library to obey robots.txt
Ruby
Latest commit 99b8962 Jan 20, 2012 @chriskite initial commit
Failed to load latest commit information.
lib
spec
.gitignore
CHANGELOG.rdoc
Gemfile
LICENSE
README.rdoc
Rakefile
VERSION
robotex.gemspec

README.rdoc

Robotex

Obey Robots.txt

With one line of code, Robotex (pronounced like “robotics”) will download and parse the robots.txt file and let you know if your program is allowed to visit a given link.

Usage:

robotex = Robotex.new "My User Agent"
robotex.allowed?("http://www.example.com/foo")
robotex.delay! # wait until any specified Crawl-Delay has passed

Acknowledgements

Robotex is a modified version of Kyle Maxwell's excellent Robots library. Some folks were unable to use that gem due to packaging issues, so I used his code to create Robotex.