Support the Sitemap extension in robotparser

[rhettinger]

BPO	21475
Nosy	@rhettinger, @ned-deily, @berkerpeksag, @matrixise, @mcscope
PRs	bpo-21475: Support the Sitemap extension in robotparser #6883
Dependencies	bpo-25497: Rewrite test_robotparser
Files	robotparser_site_maps_v1.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2018-05-16.14:54:05.895>
created_at = <Date 2014-05-12.01:35:57.203>
labels = ['easy', '3.8', 'type-feature', 'library']
title = 'Support the Sitemap extension in robotparser'
updated_at = <Date 2018-05-16.14:54:05.893>
user = 'https://github.com/rhettinger'

bugs.python.org fields:

activity = <Date 2018-05-16.14:54:05.893>
actor = 'ned.deily'
assignee = 'none'
closed = True
closed_date = <Date 2018-05-16.14:54:05.895>
closer = 'ned.deily'
components = ['Library (Lib)']
creation = <Date 2014-05-12.01:35:57.203>
creator = 'rhettinger'
dependencies = ['25497']
files = ['40791']
hgrepos = []
issue_num = 21475
keywords = ['patch', 'easy']
message_count = 14.0
messages = ['218308', '218318', '252528', '253027', '253035', '253063', '255225', '255228', '311416', '315362', '316740', '316743', '316811', '316813']
nosy_count = 7.0
nosy_names = ['rhettinger', 'ned.deily', 'berker.peksag', 'matrixise', 'pwirtz', 'stevensalbert', 'mcscope@gmail.com']
pr_nums = ['6883']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue21475'
versions = ['Python 3.8']

[rhettinger]

Resources:

• http://en.wikipedia.org/wiki/Robots_exclusion_standard#Nonstandard_extensions

• https://support.google.com/webmasters/answer/183669?hl=en

• https://github.com/seomoz/reppy

[berkerpeksag]

There is a patch for Crawl-delay in bpo-16099.

[berkerpeksag]

The Crawl-delay part(bpo-16099) is now committed.

[pwirtz]

I would like to tackle this issue. Should I wait for bpo-25400 to be resolved first?

[berkerpeksag]

bpo-25400 is not a blocker of this, so feel free to write a patch.

[pwirtz]

Here is a patch that provides support for the Sitemap extension.

[matrixise]

Add a test with your patch.

Thank you

[berkerpeksag]

Peter didn't write a test because bpo-25497 (test_robotparser rewrite) needs to be committed first. See msg253016 in bpo-25400 for more information.

[matrixise]

Hi @berker and @pwirtz.

could you write a test for this issue?

thanks

[stevensalbert]

Kindly add a test for this issue