This is a template for simple web crawler written in Python. You just provide starting URL and it crawls through all links and sublinks (with any nested level) found on pages. As a developer you can extend its functionality. Several ideas:
- Download images found on web pages.
- Extract email addresses.
- Count specific keywords.
- etc.
Folder examples/ contains some code which demonstrates how to use Green Crawler.
More detailed documentation is coming soon...