Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP
Robotexclusionrulesparser is an alternative to the Python standard library module robotparser.
Python
branch: master

Fetching latest commit…

Cannot retrieve the latest commit at this time

Failed to load latest commit information.
INSTALL
LICENSE
README
ReadMe.html
VERSION
parser_test.py
robotexclusionrulesparser.py
setup.py

README

Robotexclusionrulesparser is an alternative to the Python standard library
module robotparser. It fetches and parses robots.txt files and can answer
questions as to whether or not a given user agent is permitted to visit a 
certain URL.

This module has some features that the standard library module robotparser 
does not, including the ability to decode non-ASCII robots.txt files, respect
for Expires headers and understanding of Crawl-delay and Sitemap directives 
and wildcard syntax in path names.

Complete documentation (including a comparison with the standard library
module robotparser) is available in ReadMe.html.

Robotexclusionrulesparser is released under a BSD license. 
Something went wrong with that request. Please try again.