Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP
Ruby library to obey robots.txt
Ruby
Branch: master
Failed to load latest commit information.
lib initial commit
spec initial commit
.gitignore initial commit
CHANGELOG.rdoc initial commit
Gemfile initial commit
LICENSE initial commit
README.rdoc initial commit
Rakefile initial commit
VERSION initial commit
robotex.gemspec initial commit

README.rdoc

Robotex

Obey Robots.txt

With one line of code, Robotex (pronounced like “robotics”) will download and parse the robots.txt file and let you know if your program is allowed to visit a given link.

Usage:

robotex = Robotex.new "My User Agent"
robotex.allowed?("http://www.example.com/foo")
robotex.delay! # wait until any specified Crawl-Delay has passed

Acknowledgements

Robotex is a modified version of Kyle Maxwell's excellent Robots library. Some folks were unable to use that gem due to packaging issues, so I used his code to create Robotex.

Something went wrong with that request. Please try again.