PHP script to crawl specific domains or several domains recursively for user specified data
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
css
js
README
backend.php
crawler.php
index.php
stream.php

README

PHP script to crawl a specific website, domain or several domains recursively for user specified data.

Currently crawls for e-mail addresses only, but more content types will be added later on.

TODO:
=====

Begun:
* Create a basic frontend for setting crawl location and behavior
* Use frontend to "live broadcast" what's being crawled

Later:
* Respect robots.txt files