This project uses node.js to ingest into Apache Accumulo via RabbitMQ and a Java application. The intent is to store records of a client visiting a website, much like a Google Analytics. This example is meant to be contrived and the intent is to show the potential of using node to (indirectly) ingest data into Apache Accumulo.
node.js runs an HTTP webserver which accepts incoming requests from an HTTP client. Upon receipt, it strips the intended information off the query string of the request URL and fires a JSON string over RabbitMQ. Meanwhile, a Java process is running in the background, pulling data off of a queue that the node server is writing to. Upon receipt of a message, the Java process converts the JSON string into an object, creates, and then inserts mutations into Accumulo corresponding to the data contained in the JSON string.
RabbitMQ and node.js
These should be relatively straightfoward, something along the lines of:
# emerge net-misc/rabbitmq-server net-libs/nodejs
# /etc/init.d/rabbitmq start
Running the application
Make sure you have Hadoop, Zookeeper, and Accumulo started, then install the dependencies for the application and start it.
$ cd node $ npm install $ NODE_ENV=production node node/server.js
Build and run the AmqpWebAnalytics class
$ cd java/webanalytics $ mvn package $ bin/run.sh
Fire up curl to load some data
$ curl http://localhost:3000/analytics/10.0.0.1 -d "visitor=10.0.0.2" $ curl http://localhost:3000/analytics/10.0.0.1 -d "visitor=10.0.0.3" $ curl http://localhost:3000/analytics/10.0.0.1 -d "visitor=10.0.0.4"
Use the site to view the data at http://localhost:3000. You can also open up the Accumulo shell to verify that the record exists.
root@accumulo analytics> scan 10.0.0.1 10.0.0.2:1335928232348  10.0.0.1 10.0.0.3:1335928232353  10.0.0.1 10.0.0.4:1335928232358 
If you really want to test out the server, try running siege to flood the process :D
# emerge -av app-benchmarks/siege $ siege.config $ siege -b -c 800 "http://localhost:3000/analytics/10.0.0.1 POST visitor=10.0.0.2"