spider in node.js
1 can set how many request per second
2 can use extend to inherit
npm install xspider
`js
var Spider = require('xspider').Spider,
Crawler = require('xspider').Crawler;
var s = new Spider('http://www.sina.com.cn/'),
see examples/v2ex.js
create a spider with option:
var s = new Spider(option)
options: maxConnections: max connection conncurry rps: max requests per second, if less than 1, for example 0.5, the spider request per two seconds. maxPages: max pages to be crawled.
start(crawler)
start spider to crawl. before start, you should set the spider's crawler instance, use crawler() method or call start with a crawler instance.
stop()
stop spider.
cycle(crawler, interval)
crawler(crawl)
set or get the spider's crawler instance
pause()
pause spider.
resume()
resume a paused spider.
extend the Spider class
`js
var MySpider = Spider.extend({
start: ....
})
set crawler callbacks for specify urls.
router(url)
return a crawler's method to handle this url. crawler has two basic method: index, detail.
fetch(url)
internal used. this method return a promise instance.
handle(url)
this method is an interface to handle a url. internally, it first call router to find if there is a method to handle this url, if found, it first call fetch method, then use the method to handle to url and it's response.
extend the Crawler class
`js
var MyCrawler = Crawler.extend({
index: function(url, resp) { return [] },
detail: function(url, resp) {return [] },
})
the callback function should return an array, which is an url list to be crawled.