Permalink
Browse files

fixed error that got silenced and broke whole library. rescue nil mig…

…ht not be the right way to go here...
  • Loading branch information...
1 parent 69c6160 commit 8ab1b10c03156ae809c9284ea39ab404dd35d134 @rb2k rb2k committed Jun 23, 2010
Showing with 2 additions and 2 deletions.
  1. +2 −2 lib/robots.rb
View
@@ -12,7 +12,7 @@ class ParsedRobots
def initialize(uri, user_agent)
@last_accessed = Time.at(1)
- io = Robots.get_robots_txt(uri)
+ io = Robots.get_robots_txt(uri, user_agent)
if !io || io.content_type != "text/plain" || io.status != ["200", "OK"]
io = StringIO.new("User-agent: *\nAllow: /\n")
@@ -98,7 +98,7 @@ def to_regex(pattern)
end
end
- def self.get_robots_txt(uri)
+ def self.get_robots_txt(uri, user_agent)
begin
Timeout::timeout(Robots.timeout) do
io = URI.join(uri.to_s, "/robots.txt").open("User-Agent" => user_agent) rescue nil

0 comments on commit 8ab1b10

Please sign in to comment.