Spidey: A Focused Web Crawler
A focused web crawler using Python
Web crawling is the first step in buildling a web search engine. It refers to browsing the web in a methodical, automated manner, with the aim of downloading pages to be indexed. Focused crawling is a special type of crawling wherein only pages that are related to a specific topic are crawled.
Read readme.txt and explain.txt for more details.