Riak data as input to hadoop m/r and output of hadoop m/r
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Failed to load latest commit information.



Travis-CI :: https://secure.travis-ci.org/basho/riak-hadoop.png

Riak-Hadoop is a library for using Riak as an input/output to Hadoop Map/Reduce.

NOTE This library is not yet offically supported by Basho and is strictly experimental.

How it works

The library extends InputFormat, InputSplit, RecordReader, OutputFormat and RecordWriter, so you can declare any valid Riak M/R input (2i query, riak search query, list of keys, bucket) as input to a Hadoop M/R job. The library will split the keys into partitions and Hadoop will use the RiakRecordReader to load Key/Value pairs from Riak for the Mapper tasks. The Reducer output is written back to a configured bucket in Riak. It uses the Riak-Java-Client to talk to Riak. You just write a normal Hadoop Map/Reduce job, but declare RiakInputFormat and RiakOutpurFormat as sources/sinks for your data/results.


Have a look at the Riak Word Count example project to get started with Riak-Hadoop.


I plan to have the library load bulk data from Riak to HDFS to better leverage Hadoop’s integration with that file system. There will be docs.


Raise issues, pull requests, comments etc. as ever through GitHub, or the Riak mailing list.