Build and Push Jobs for Voldemort Read Only Stores

abh1nay edited this page Oct 24, 2012 · 5 revisions


We have been using the Build and Push Job at Linkedin to create Voldemort Read-Only stores from data present in sequence files/ Avro Container files on HDFS.

The VoldemortBuildAndPushJob will behave in the following way:

  1. Build an XML storeDef for your data (based off of the key and value metadata in your JsonSequenceFile/Avro on HDFS).

  2. Connect to push.cluster

  3. Get the storeDefs for all stores in the push.cluster

  4. Look through the storeDefs for a store with the same name as If one is found, validate that the storeDef in the cluster matches the storeDef for your data. If it doesn’t, fail. If no storeDef exists on the cluster that matches, then add your storeDef to the cluster.

  5. Build the Voldemort store in Hadoop

  6. Push the Voldemort store to push.cluster

Pushing JSON Data

push.cluster=tcp://localhost:6666"test store"

Pushing AVRO Data

azkaban.should.proxy=true anagpal
build.replication.factor= 1
avro.value.field=localizedFirstNames"Testing avro build and push"

Notice the following properties:

1) build.type.avro=true This specifies that input data is Avro

2) avro.key.field=memberId This specifies the field to be used as the key

3) avro.value.field=localizedFirstNames This specifies the field to be used as the value

Properties description

The properties listed in the job above are required, but there are additional properties that you may specify:

Property	         |        Required    |               Description	         |         Y	    | The name of the store that the job will try to push.
push.cluster	         |         Y	    | The cluster URI to push to. For example, tcp://localhost:6666.
build.input.path         |	   Y	    | The Hadoop data that you wish to push.
                         |                  | This data must be in Avro/JsonsequenceFile format
build.output.dir         |         Y	    | The Hadoop directory where the Voldemort store will be stored        |	   Y	    | The owner email ID.   | 	   Y	    | The service name that uses this store and usage in brief.
type	                 |         Y        |  java for our use case
num.chunks	         |         N	    |  Number of chunks per partition.
build.replication.factor |	   N	    |The replication factor you want this store to have.
build.compress.value	 |         N	    |If present, compresses each value using gzip
build.temp.dir	         |         N	    |The Hadoop directory where temporary data should be stored.
                         |                  |Temp data is generated when the store is built on HDFS.
build.chunk.size	 |         N	    | The individual chunk size (advisory).