Permalink
Browse files

Don't parse HEAD of robots.txt responses.

  • Loading branch information...
1 parent 6b10451 commit beb25d185ef5385af9afb3209176009eb5410d81 @scop scop committed Apr 30, 2011
Showing with 2 additions and 0 deletions.
  1. +2 −0 lib/LWP/RobotUA.pm
View
@@ -126,7 +126,9 @@ sub simple_request
$self->{'rules'}->parse($robot_url, "");
my $robot_req = HTTP::Request->new('GET', $robot_url);
+ my $parse_head = $self->parse_head(0);
my $robot_res = $self->request($robot_req);
+ $self->parse_head($parse_head);
my $fresh_until = $robot_res->fresh_until;
my $content = "";
if ($robot_res->is_success && $robot_res->content_is_text) {

1 comment on commit beb25d1

@putaotao

This change will can't parse the real request HEAD.
The response can't get base url when the html source code include ''.

Please sign in to comment.