Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP
Browse files

fixed error that got silenced and broke whole library. rescue nil mig…

…ht not be the right way to go here...
  • Loading branch information...
commit 8ab1b10c03156ae809c9284ea39ab404dd35d134 1 parent 69c6160
@rb2k rb2k authored
Showing with 2 additions and 2 deletions.
  1. +2 −2 lib/robots.rb
View
4 lib/robots.rb
@@ -12,7 +12,7 @@ class ParsedRobots
def initialize(uri, user_agent)
@last_accessed = Time.at(1)
- io = Robots.get_robots_txt(uri)
+ io = Robots.get_robots_txt(uri, user_agent)
if !io || io.content_type != "text/plain" || io.status != ["200", "OK"]
io = StringIO.new("User-agent: *\nAllow: /\n")
@@ -98,7 +98,7 @@ def to_regex(pattern)
end
end
- def self.get_robots_txt(uri)
+ def self.get_robots_txt(uri, user_agent)
begin
Timeout::timeout(Robots.timeout) do
io = URI.join(uri.to_s, "/robots.txt").open("User-Agent" => user_agent) rescue nil
Please sign in to comment.
Something went wrong with that request. Please try again.