Switch branches/tags
Nothing to show
Find file History
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
..
Failed to load latest commit information.
README.md
crawl_a_list.js
crawl_the_web.js
urllist.txt

README.md

openwebspider(js) sample tools

Sample tools that take advantage of the indexer web service

crawl_a_list

Crawls all the URLs in a file

$ node crawl_a_list.js urllist.txt

crawl_the_web

Starts from an URL and crawls all the pages found

$ node crawl_the_web.js http://www.example.com/