fizx / robots

robots.txt parser

This URL has Read+Write access

better
Kyle Maxwell (author)
Tue Mar 24 15:11:13 -0700 2009
commit  f3cdfc5020785e4e34654a29b2e75b5b9ca70b5b
tree    6c9d0f1da7c5953f46545e91b66b12097298100c
parent  d45462fd2f26e9e1a14986e04da28cb498af80e1
robots / CHANGELOG
raw normal view history
b1543c9f » Kyle Maxwell 2008-12-10 0.4 1 0.4.0
2 - Fixed other_values bug
3 - added crawl-delay support
5f6ab6ba » Kyle Maxwell 2008-11-25 test more robots 4 0.3.2
5 - fixed breaking on reddit.com
1f0a1b0f » Kyle Maxwell 2008-11-05 v0.3.1 6 0.3.1
7 - fixed bug in disallows handling
8 - partially mocked out open-uri
3d449111 » Kyle Maxwell 2008-09-26 loggable 9 0.3.0
10 - added loggable dependency
e643bdbe » Kyle Maxwell 2008-09-23 what if robots 404s 11 0.2.0
12 - IF robot.txt 404s, assume allowed.
8cf93bc5 » Kyle Maxwell 2008-12-10 merged 13 - Added CHANGELOG