*Deprecated* A short and sweet Python web crawler using Redis as the process queue, seen set and Memcache style rate limiter for robots.txt
Switch branches/tags
Nothing to show
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Failed to load latest commit information.



As the world doesn't have enough horrible crawlers

This is a minimalist web crawler written in Python for fun over two evenings. PubCrawl currently uses Redis as a persistent backend for the URL frontier (queue), seen URL sets and Memcache style key expiration for use with robots.txt. On a commodity laptop with a standard ADSL2+ connection the download rate is a sustained 25-50 pages per second (or 2-4 million crawled pages per day).

  • Multiple Python clients can be started and run at the same time either using multiple processes (multiprocessing) or threads
  • The crawl can be stopped and restarted with only a minimal loss of URLs (zero by the future addition of an in-progress set of URLs)
  • The web crawler respects robots.txt even when the crawl is stopped and restarted (as long as the Redis database layer persists)
  • PubCrawl only depends on Redis which plays the role of message queue, URL set curator and Memcache server

The web crawler respects robots.txt through the use of a slightly modified robot exclusion rules parser by Philip Semanchuk. Currently when a website is retrieved the web domain is added as a key to Redis and given an expiry time. The expiry time is either the one provided by robots.txt or a one second program default.

For actual production use it's suggested to install a local DNS caching server such as dnsmasq or ncsd for performance reasons.


  • Python 2.6 (due to multiprocessing)
  • Redis
  • redis-py sudo easy_install redis


  • Investigate alternative methods for fetching URLs in order to improve the speed and concurrency over urllib2.urlopen
  • Better busy queue implementation for handling links that are constrained due to the robots.txt delay
  • The link graph may be better stored on disk (or at the very least there should be an interface for storage/manipulation)
  • Implement a flat file storage system for the page contents (as this is currently only useful for the link graph)
  • The cache for robots.txt should be globally accessible and not a localised Python object
  • Allow for modifying both the structure, database layer and location of the Redis database server
  • For longer crawls exceptions and their tracebacks are ignored but should be stored for later review
  • Make CrawlRequest easily extendible so that a new layer of processing can be added arbitrarily
  • Improve the robots.txt parser to handle Unicode and generally more complex formats