Robotexclusionrulesparser is an alternative to the Python standard library module robotparser.
Python
Switch branches/tags
Nothing to show
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
INSTALL
LICENSE
README
ReadMe.html
VERSION
parser_test.py
robotexclusionrulesparser.py
setup.py

README

Robotexclusionrulesparser is an alternative to the Python standard library
module robotparser. It fetches and parses robots.txt files and can answer
questions as to whether or not a given user agent is permitted to visit a 
certain URL.

This module has some features that the standard library module robotparser 
does not, including the ability to decode non-ASCII robots.txt files, respect
for Expires headers and understanding of Crawl-delay and Sitemap directives 
and wildcard syntax in path names.

Complete documentation (including a comparison with the standard library
module robotparser) is available in ReadMe.html.

Robotexclusionrulesparser is released under a BSD license.