simple web crawler with node. experimenting with streams.
wubwub.crawl({
'routes': {
//Routes that get called and crawling does not progress beyond
'leaf': {
'.*en.wikipedia.org/wiki/.*': function(tr, link) {
tr.selectAll('p', function(div) {
div.createReadStream().pipe(process.stdout);
});
},
},
//Routes that get called and all links are enqueued
'tree': {
'.*en.wikipedia.org/wiki/.*': function(tr, link) {}
},
//Routes not to follow
'ignore': [],
},
'seed': ['http://en.wikipedia.org/wiki/Earth'],
'concurrency': 50,
'backend': new wubwub.Backends.Simple()
});
##wat? This example will start at wikipedia.org, and follow any link and print the contents of every p element on every leaf page to stdout
There are three kinds of routes, leaf routes, tree routes, and ignored routes.
- Ignored routes: a regex in the ignored route list won't be fetched. This overrides any previous route spec, such that if a leaf and ignored route are matched, the route is ignored.
- Leaf routes: a leaf route is one that when matched, will call the callback specified but not enqueue any of the links on the page.
- Tree route: a leaf route is one that when matched, will call the callback specified and enqueue all the links on the page (provided they aren't ignored and match one of the tree or leaf routes)
Callbacks in the leaf and tree routes are passed a trumpet instance for the page that was just fetched, as well as the url that was matched.
Specifies how many requests can be running at the same time
Specifies which URL to start crawling at
Controls how links are enqueued for crawling. Two are included, wubwub.Backends.Simple() and wubwub.Backends.Redis() which implement put() and get(onLink) methods for links, and keep track of what URLs have already been seen. You can implement your own backend as long as it implements those methods. get() is async and takes a callback to be executed once the link is fetched.