A web Spider (Crawler) that follows all the links with a domain name
JavaScript
Switch branches/tags
Nothing to show
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
deps
lib
.gitignore
.gitmodules
README.md
designblogs.txt
spider-test.js

README.md

A Node Web Spider

This is a simple node spider that crawls all of the links within a domain.

Example

The example crawls follows of the links of a given website domain and downloads the text of each page. Using a email regex the example searches for email addresses on the page.

run:

$node spider-test.js

to try it out.