Skip to content
Browse files

initial successful commit

  • Loading branch information...
1 parent 084cfc4 commit 730bb40607a583e78bfee32a2bf7a24b95a4bb07 @fizx committed Aug 10, 2008
Showing with 55 additions and 0 deletions.
  1. +1 −0 .gitignore
  2. +33 −0 README
  3. +7 −0 { → lib}/robots.rb
  4. +14 −0 robots.gemspec
View
1 .gitignore
@@ -0,0 +1 @@
+*.gem
View
33 README
@@ -0,0 +1,33 @@
+A simple Ruby library to parse robots.txt.
+
+Usage:
+
+ robots = Robots.new "Some User Agent"
+ assert robots.allowed?("http://www.yelp.com/foo")
+ assert !robots.allowed?("http://www.yelp.com/mail?foo=bar")
+ robots.other_values # gets misc. key/values (i.e. sitemaps)
+
+If you want caching, you're on your own. I suggest marshalling an instance of the parser.
+
+Copyright (c) 2008 Kyle Maxwell
+
+Permission is hereby granted, free of charge, to any person
+obtaining a copy of this software and associated documentation
+files (the "Software"), to deal in the Software without
+restriction, including without limitation the rights to use,
+copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the
+Software is furnished to do so, subject to the following
+conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES
+OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
+HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
+WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+OTHER DEALINGS IN THE SOFTWARE.
View
7 robots.rb → lib/robots.rb
@@ -92,6 +92,13 @@ def allowed?(uri)
@parsed[host] ||= ParsedRobots.new(uri)
@parsed[host].allowed?(uri, @user_agent)
end
+
+ def other_values(uri)
+ uri = URI.parse(uri.to_s) unless uri.is_a?(URI)
+ host = uri.host
+ @parsed[host] ||= ParsedRobots.new(uri)
+ @parsed[host].other_values
+ end
end
if __FILE__ == $0
View
14 robots.gemspec
@@ -0,0 +1,14 @@
+Gem::Specification.new do |s|
+ s.name = "robots"
+ s.version = "0.1"
+ s.date = "2008-08-10"
+ s.summary = "Simple robots.txt parser"
+ s.email = "kyle@kylemaxwell.com"
+ s.homepage = "http://github.com/fizx/robots"
+ s.description = "It parses robots.txt files"
+ s.has_rdoc = true
+ s.authors = ["Kyle Maxwell"]
+ s.files = ["README", "lib/robots.rb"]
+ s.rdoc_options = ["--main", "README"]
+ s.extra_rdoc_files = ["README"]
+end

0 comments on commit 730bb40

Please sign in to comment.
Something went wrong with that request. Please try again.