Web crawler in Node.js
JavaScript CSS Shell
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
api Do not delete crawl data (in addition to nodes) Dec 25, 2013
cli Drop old code Nov 20, 2013
crawler
web Fix scrolling issue Jan 7, 2014
.gitignore Merge branch 'refs/heads/master' into mongodb Nov 19, 2013
README.md Update README Nov 20, 2013

README.md

UniCrawler

There are some different parts:

Requirements

API

The API allow you to configure and control the crawling process.

Cli

Some scripts to send request to the API.

Crawler

Processing agent. You can start more agent to speed up the process.

Web

It's the WebUI. Don't forget to copy config.dist.js to config.js and setup the URL to you API.