Skip to content
Branch: master
Find file History
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
..
Failed to load latest commit information.
README.md
crawl_a_list.js
crawl_the_web.js
urllist.txt

README.md

openwebspider(js) sample tools

Sample tools that take advantage of the indexer web service

crawl_a_list

Crawls all the URLs in a file

$ node crawl_a_list.js urllist.txt

crawl_the_web

Starts from an URL and crawls all the pages found

$ node crawl_the_web.js http://www.example.com/
You can’t perform that action at this time.