Skip to content
A focused web crawler using Python
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
README.md
crawler_log_1.txt
crawler_log_2.txt
crawler_log_3.txt
crawler_log_4.txt
explain.txt
readme.txt
spidey.py
spidey_multiThread.py

README.md

Spidey: A Focused Web Crawler

A focused web crawler using Python

Web crawling is the first step in buildling a web search engine. It refers to browsing the web in a methodical, automated manner, with the aim of downloading pages to be indexed. Focused crawling is a special type of crawling wherein only pages that are related to a specific topic are crawled.

Read readme.txt and explain.txt for more details.

You can’t perform that action at this time.