NOTE This library is not yet offically supported by Basho and is strictly experimental.
How it works
The library extends
RecordWriter, so you can declare any valid Riak
M/R input (2i query, riak search query, list of keys, bucket) as input
to a Hadoop M/R job. The library will split the keys into partitions
and Hadoop will use the
RiakRecordReader to load Key/Value pairs
from Riak for the
Mapper tasks. The
Reducer output is written back
to a configured bucket in Riak. It uses the Riak-Java-Client to talk
to Riak. You just write a normal Hadoop Map/Reduce job, but declare
RiakOutpurFormat as sources/sinks for your
Have a look at the Riak Word Count example project to get started with Riak-Hadoop.
I plan to have the library load bulk data from Riak to HDFS to better leverage Hadoop’s integration with that file system. There will be docs.
Raise issues, pull requests, comments etc. as ever through GitHub, or the Riak mailing list.