Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP
Browse files

Decode robots.txt response content before attempting to parse it.

  • Loading branch information...
commit 83be3c5e7a9fb0c4b972008aa38e8a5bbad7f655 1 parent 38ea477
@scop scop authored
Showing with 2 additions and 2 deletions.
  1. +2 −2 lib/LWP/RobotUA.pm
View
4 lib/LWP/RobotUA.pm
@@ -129,8 +129,8 @@ sub simple_request
my $robot_res = $self->request($robot_req);
my $fresh_until = $robot_res->fresh_until;
if ($robot_res->is_success && $robot_res->content_is_text) {
- my $c = $robot_res->content;
- if ($c =~ /^\s*Disallow\s*:/mi) {
+ my $c = $robot_res->decoded_content;
+ if ($c && $c =~ /^\s*Disallow\s*:/mi) {
$self->{'rules'}->parse($robot_url, $c, $fresh_until);
}
else {
Please sign in to comment.
Something went wrong with that request. Please try again.