"A fun toy"
Just a sample crawler to tell you if you have any dead links. Please keep in mind this is just an off-the-cuff project and not meant to represent any sort of good practices.
git clone git://github.com/sgrove/nodespider.git
npm install request
npm install jsdom
node crawl.js http://news.ycombinator.com
- Add credentials for basic auth to be able to crawl behind protected sites as well.
- Setup connection pooling
- Make it "work" :)
Vekz - For his node mastery and help