Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP
Branch: master
Fetching contributors…

Cannot retrieve contributors at this time

27 lines (26 sloc) 0.64 kB
0.10.0
- Make sure the fetch robots.txt operation happens with a user agent (via rb2k)
0.9.0
- Fix http://github.com/fizx/robots/issues#issue/1
- Tests don't rely on network.
0.8.0
- Add multiple values from robots.txt (via joost)
0.7.3
- Move to jeweler, gemcutter
0.7.2
- Add Ruby 1.9 compatibility
0.5-0.7.1
- Lost the changelog information :/
0.4.0
- Fixed other_values bug
- added crawl-delay support
0.3.2
- fixed breaking on reddit.com
0.3.1
- fixed bug in disallows handling
- partially mocked out open-uri
0.3.0
- added loggable dependency
0.2.0
- IF robot.txt 404s, assume allowed.
- Added CHANGELOG
Jump to Line
Something went wrong with that request. Please try again.