Skip to content
things uses with hbase
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Failed to load latest commit information.


== lasthbase ==

A java library uses with hbase. Currently just table input and output
formats for using dumbo with hbase.
 - HBase:
 - Dumbo:

=== Using dumbo over HBase ===

These assume you are storing everything in hbase as if they are byte strings.
You should also note, the tables and families you are writing and reading to
must already exist in hbase.

The input format, will give key values to your mapper as:
(row, {family: {qualifier1: value, qualifier2 : value2}, family2: {qualifier3: value3} })

The output format takes the same format. The row, families, qualifiers, and 
values must all be strings (for now). 

To use:
1. you obviously must already have hadoop and hbase setup so you can run java 
   mapreduce jobs over hbase first (have hbase jar in hadoop lib folder, etc).
2. build the lasthbase.jar, with ant. This project is not using a release version of hbase yet, replace the jars with the hbase.jar you're using before compiling.
3. write your dumbo job

eg. using the input format 

import dumbo

def mapper(key, columns):
    for family in columns:
        for qualifier, value in columns[family].iteritems():
            yield key, (family, qualifier, value)

def runner(job):

if __name__ == "__main__":

eg. using the output format.

import dumbo

def mapper(key, column):
    columns = {}
    for family, qualifier, value in column:
        column = columns.get(family, {})
        column[qualifier] = value
        columns[family] = column
    yield key, columns

def runner(job):

if __name__ == "__main__":

4. Starting your dumbo job over hbase:

$ dumbo -hadoop <hadoopdir> -libjar lasthbase.jar \
-inputformat fm.last.hbase.mapred.TypedBytesTableInputFormat \
-hadoopconf hbase.mapred.tablecolumns="family1:qualifier1 family1:qualifier2 family2:qualifier3" \
-input input_table -output output_dir

$ dumbo -hadoop <hadoopdir> -libjar lasthbase.jar \ 
-outputformat fm.last.hbase.mapred.TypedBytesTableOutputFormat \
-input test_data \
-jobconf hbase.mapred.outputtable=output_table \
-output ignoresthisbutyouneeditsorry

Something went wrong with that request. Please try again.