.gitignore lib/robots.rb robots.gemspec @@ -0,0 +1,33 @@ +A simple Ruby library to parse robots.txt. + +Usage: + + robots = Robots.new "Some User Agent" + assert robots.allowed?("http://www.yelp.com/foo") + assert !robots.allowed?("http://www.yelp.com/mail?foo=bar") + robots.other_values # gets misc. key/values (i.e. sitemaps) + +If you want caching, you're on your own. I suggest marshalling an instance of the parser. + +Copyright (c) 2008 Kyle Maxwell + +Permission is hereby granted, free of charge, to any person +obtaining a copy of this software and associated documentation +files (the "Software"), to deal in the Software without +restriction, including without limitation the rights to use, +copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the +Software is furnished to do so, subject to the following +conditions: + +The above copyright notice and this permission notice shall be +included in all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, +EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES +OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND +NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT +HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, +WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING +FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +OTHER DEALINGS IN THE SOFTWARE. \ No newline at end of file README robots.rb 084cfc417360920eba5061c8c7c35192594ca1da Kyle Maxwell kyle@kylemaxwell.com http://github.com/fizx/robots/commit/730bb40607a583e78bfee32a2bf7a24b95a4bb07 730bb40607a583e78bfee32a2bf7a24b95a4bb07 2008-08-10T23:30:30-07:00 2008-08-10T23:30:30-07:00 initial successful commit 2331d5bfddb46b25eaf5ae33c7e7974e59ae404f Kyle Maxwell kyle@kylemaxwell.com