Skip to content


Subversion checkout URL

You can clone with
Download ZIP
Scrapy project to scrape public web directories (educational)
branch: master

This branch is 23 commits behind scrapy:master

Fetching latest commit…

Cannot retrieve the latest commit at this time

Failed to load latest commit information.



This is a Scrapy project to scrape websites from public web directories.

This project is only meant for educational purposes.


The items scraped by this project are websites, and the item is defined in the class:


See the source code for more details.


This project contains two spiders: googledir and dmoz. When in doubt, you can check the available spiders with:

scrapy list

Spider: googledir

The googledir spider crawls the entire Google Directory, though you may want to try it by limiting the crawl to a certain number of items.

For example, to run the googledir spider limited to scrape 20 items use:

scrapy crawl googledir --set CLOSESPIDER_ITEMCOUNT=20

Spider: dmoz

The dmoz spider scrapes the Open Directory Project (, and it's based on the dmoz spider described in the Scrapy tutorial

Unlike the googledir spider, this spider doesn't crawl the entire site but only a few pages by default (defined in the start_pages attribute). These pages are:

So, if you run the spider regularly (with scrapy crawl dmoz) it will scrape only those two pages. However, you can scrape any page by passing the url instead of the spider name. Scrapy internally resolves the spider to use by looking at the allowed domains of each spider.

For example, to scrape a different URL use:

scrapy crawl

You can scrape any URL from using this spider

  • googledir - for scraping Google Directory (


This project uses a pipeline to filter out websites containing certain forbidden words in their description. This pipeline is defined in the class:

Something went wrong with that request. Please try again.