Add support for crawling HTML fragments

[jhurliman]

If the crawler encounters a page such as http://hypem.com/blog/a/1?ax=1 that contains an HTML fragment and not a complete document, the crawler will crash when trying to appendChild() a jQuery script element on the undefined window.document.body object. One option might be to add support for the HTML5 parser library which should be able to handle fragments: https://github.com/aredridel/html5

A short repro:

var Crawler = require('node-crawler').Crawler;
var crawler = new Crawler({
  maxConnections: 1,
  callback: function(err, res, $) {
    console.log('worked!'); // The app will crash before this point
  }
});

crawler.queue(['http://hypem.com/blog/a/1?ax=1']);

[jhurliman]

I apologize, I was trying to run node-crawler with a more recent version of jsdom (0.2.0) which seemed to be causing this issue.