Robotparser does not handle empty paths

[cmalamas]

BPO	522898
Nosy	@loewis

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2002-02-28.15:32:04.000>
created_at = <Date 2002-02-26.10:40:51.000>
labels = ['library']
title = 'Robotparser does not handle empty paths'
updated_at = <Date 2002-02-28.15:32:04.000>
user = 'https://bugs.python.org/cmalamas'

bugs.python.org fields:

activity = <Date 2002-02-28.15:32:04.000>
actor = 'loewis'
assignee = 'none'
closed = True
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2002-02-26.10:40:51.000>
creator = 'cmalamas'
dependencies = []
files = []
hgrepos = []
issue_num = 522898
keywords = []
message_count = 2.0
messages = ['9423', '9424']
nosy_count = 2.0
nosy_names = ['loewis', 'cmalamas']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = None
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue522898'
versions = []

[cmalamas]

The robotparser module handles incorrectly empty paths
in the allow/disallow directives.

According to: http://www.robotstxt.org/wc/norobots-
rfc.html, the following rule should be a global
*allow*:
User-agent: *
Disallow:

My reading of the RFC is that an empty path is always
a global allow (for both Allow and Disallow
directives) so that the syntax is backwards
compatible --there was no Allow directive in the
original syntax.

Suggested fix:
robotparser.RuleLine.applies_to() becomes:
def applies_to(self, filename):
if not self.path:
self.allowance = 1
return self.path=="*" or re.match(self.path,
filename)

[loewis]

Logged In: YES
user_id=21627

This is fixed in robotparser.py 1.11.