fizx / robots

robots.txt parser

This URL has Read+Write access

better
Kyle Maxwell (author)
Tue Mar 24 15:11:13 -0700 2009
commit  f3cdfc5020785e4e34654a29b2e75b5b9ca70b5b
tree    6c9d0f1da7c5953f46545e91b66b12097298100c
parent  d45462fd2f26e9e1a14986e04da28cb498af80e1
robots / CHANGELOG
100644 14 lines (13 sloc) 0.279 kb
raw blame history 
1
2
3
4
5
6
7
8
9
10
11
12
13
14

0.4.0 
  - Fixed other_values bug
  - added crawl-delay support
0.3.2
	- fixed breaking on reddit.com
0.3.1
  - fixed bug in disallows handling
  - partially mocked out open-uri
0.3.0
  - added loggable dependency 
0.2.0
	- IF robot.txt 404s, assume allowed.
  - Added CHANGELOG