Skip to content
This repository

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP
branch: master
Fetching contributors…

Octocat-spinner-32-eaf2f5

Cannot retrieve contributors at this time

file 30 lines (17 sloc) 1.753 kb

RedisStorer

A UDF StoreFunc for Apache Pig designed to bulk-load data into Redis. Inspired by wonderdog, the Infochimps bulk-loader for elasticsearch.

Compiling and running

Compile:

Dependencies are automatically retrieved using Ivy.

$ ant hadoop

Use:

$ pig
grunt> REGISTER dist/pig-redis.jar;
grunt> a = LOAD 'somefile.tsv' USING PigStorage('\t');
grunt> STORE a INTO 'dummy-filename-is-ignored' USING com.hackdiary.pig.RedisStorer('kv', 'localhost');

Bulkloading strategy

RedisStorer runs in four modes: kv, set, hash and list (specified as the first argument to RedisStorer). If no mode is specified, kv is the default.

In kv mode, it takes the first field of the stored tuple as the key, and the second field as the value, and issues SET key value. Any further fields are ignored.

In set mode, it takes the first field of the stored tuple as the key, and issues SADD key value once for each subsequent field value in the tuple.

In hash mode, it takes the first field of the stored tuple as the key, and issues HSET key fieldname value once for each subsequent field value, using the same key for each, and taking the fieldname from the tuple's schema fieldnames. This means that it will fail unless the stored tuple has a schema with named fields.

In list mode, it takes the first field of the stored tuple as the key, and issues LPUSH key value once for each subsequent field value in the tuple.

Something went wrong with that request. Please try again.