This is a Scrapy project to scrape websites from public web directories.
This project is only meant for educational purposes.
The items scraped by this project are websites, and the item is defined in the class:
See the source code for more details.
This project contains one spider called
dmoz that you can see by running:
dmoz spider scrapes the Open Directory Project (dmoz.org), and it's
based on the dmoz spider described in the Scrapy tutorial
This spider doesn't crawl the entire dmoz.org site but only a few pages by
default (defined in the
start_pages attribute). These pages are:
So, if you run the spider regularly (with
scrapy crawl dmoz) it will scrape
only those two pages.
Filtering by words
A pipeline to filter out websites containing certain forbidden words in their description. This pipeline is defined in the class:
Requiring certain item fields
A pipeline to discard items that lack of certain fields. This pipeline is defined in the class:
Storing items in a MySQL database
A pipeline to store (insert or update) scraped items in a MySQL database. This pipeline is defined in the class:
The database schema is defined in
db/mysql.sql and the settings file
contains the default
MYSQL_* settings values. The scraped items will be
stored in the
website database table.
It is required to have set up the database schema before running the spider.