Ruby libraries for efficient, effective Hadoop streaming
Pull request Compare This branch is 956 commits behind infochimps-labs:master.
Failed to load latest commit information.
bin +2 for saving throws on laziness: cutc -- cuts a character range, cut… Mar 5, 2009
config Enum type, made emittable cascade its to_flat Mar 15, 2009
doc Organized to standard names Feb 14, 2009
examples word_count example assumed the presence of Enumberable#sum Apr 9, 2009
lib Script outputs the calling file's path Apr 9, 2009
.gitignore Ignore generated gems Apr 9, 2009
LICENSE.txt Added apache license. Listed off a hearty list of TODO Feb 17, 2009
README-tutorial.textile New AccumulatingReducer semantics: #start!(*args) not #reset! Mar 2, 2009
README.textile Updated readme, pulled some debug strings left in oops Feb 25, 2009
Rakefile Use Jeweler to generate an initial gemspec Apr 9, 2009
TODO.textile Update TODO Apr 9, 2009
VERSION.yml Use Jeweler to generate an initial gemspec Apr 9, 2009
wukong.gemspec Use Jeweler to generate an initial gemspec Apr 9, 2009



Wukong makes Hadoop so easy a chimpanzee can use

Treat your dataset like a

  • stream of lines when it’s efficient to process by lines
  • stream of field arrays when it’s efficient to deal directly with fields
  • stream of lightweight objects when it’s efficient to deal with objects

Wukong is friends with Hadoop the elephant,
Pig the query language, and the cat on your
command line.

How to write a Wukong script

Here’s a script to count words in a text stream:

require ‘wukong’ module WordCount class Mapper < Wukong::Streamer::LineStreamer
  1. Emit each word in the line.
    def process line
    words = line.strip.split(/\W+/).reject(&:blank?)
    words.each{|word| yield [word, 1] }
class Reducer < Wukong::Streamer::ListReducer def finalize yield [ key, ] end end end WordCount::Mapper, WordCount::Reducer ).run # Execute the script

The first class, the Mapper, eats lines and craps [word, count] records: word
is the /key/, its count is the /value/.

In the reducer, the values for each key are stacked up into a list; then the
record(s) yielded by #finalize are emitted. There are many other ways to write
the reducer (most of them are better) — see the examples

Structured data stream

You can also use structs to treat your dataset as a stream of objects:

<pre> require ‘wukong’ require ‘my_blog’ #defines the blog models
  1. structs for our input objects
    Tweet = :id, :created_at, :twitter_user_id,
    :in_reply_to_user_id, :in_reply_to_status_id, :text )
    TwitterUser = :id, :username, :fullname,
    :homepage, :location, :description )
    module TwitBlog
    class Mapper < Wukong::Streamer::RecordStreamer
  2. Watch for tweets by me
    MY_USER_ID = 24601
  3. If this is a tweet is by me, convert it to a Post.
  4. If it is a tweet not by me, convert it to a Comment that
  5. will be paired with the correct Post.
  6. If it is a TwitterUser, convert it to a User record and
  7. a user_location record
    def process record
    case record
    when TwitterUser
    user = # grab the fields in common
    user_loc =, record.location, nil, nil)
    yield user
    yield user_loc
    when Tweet
    if record.twitter_user_id == MY_USER_ID
    post = record = “{}”
    post.body = record.text
    post.title = record.text[0..65] + “…”
    yield post
    comment = record
    comment.body = record.text
    comment.post_id = record.in_reply_to_status_id
    yield comment
    end TwitBlog::Mapper, nil ).run # identity reducer

More info

There are many useful examples (including an actually-useful version of the
WordCount script) in examples/ directory.


1. Allow Wukong to discover where his elephant friend lives: either

  • set a $HADOOP_HOME environment variable,
  • or create a file ‘config/wukong-site.yaml’ with a line that points to the
    top-level directory of your hadoop install:
:hadoop_home: /usr/local/share/hadoop

2. Add wukong’s bin/ directory to your $PATH, so that you may use its
filesystem shortcuts.

How to run a Wukong script

To run your script using local files and no connection to a hadoop cluster,

your/script.rb —run=local path/to/input_files path/to/output_dir

To run the command across a Hadoop cluster,

your/script.rb —run=hadoop path/to/input_files path/to/output_dir

You can set the default in the config/wukong-site.yaml file, and then just use
--run instead of --run=something —it will just use the default run mode.

If you’re running --run=hadoop, all file paths are HDFS paths. If you’re
running --run=local, all file paths are local paths. (your/script path, of
course, lives on the local filesystem).

You can supply arbitrary command line arguments (they wind up as key-value pairs
in the options path your mapper and reducer receive), and you can use the hadoop
syntax to specify more than one input file:

./path/to/your/script.rb —any_specific_options —options=can_have_vals \ —run “input_dir/part_*,input_file2.tsv,etc.tsv” path/to/output_dir

Note that all --options must precede (in any order) all non-options.

How to test your scripts

To run mapper on its own:

cat ./local/test/input.tsv | ./examples/word_count.rb —map | more

or if your test data lies on the HDFS,

hdp-cat test/input.tsv | ./examples/word_count.rb —map | more

Next graduate to running --run=local mode so you can inspect the reducer.

What’s up with Wukong::AndPig?

Wukong::AndPig is a small library to more easily generate code for the
Pig data analysis language. See its
README for more.

Why is it called Wukong?

Hadoop, as you may know, is named after a stuffed
Since Wukong was started by the
infochimps team, we needed a simian analog. A Monkey
King who journeyed to the land of the Elephant seems to fit the bill:

Sun Wukong (孙悟空), known in the West as the Monkey King, is the main
character in the classical Chinese epic novel Journey to the West. In the novel,
he accompanies the monk Xuanzang on the journey to retrieve Buddhist sutras from

Sun Wukong possesses incredible strength, being able to lift his 13,500 jīn
(8,100 kg) Ruyi Jingu Bang with ease. He also has superb speed, traveling
108,000 li (54,000 kilometers) in one somersault. Sun knows 72 transformations,
which allows him to transform into various animals and objects; he is, however,
shown with slight problems transforming into other people, since he is unable to
complete the transformation of his tail. He is a skilled fighter, capable of
holding his own against the best generals of heaven. Each of his hairs possesses
magical properties, and is capable of transforming into a clone of the Monkey
King himself, or various weapons, animals, and other objects. He also knows
various spells in order to command wind, part water, conjure protective circles
against demons, freeze humans, demons, and gods alike. — Sun Wukong’s
Wikipedia entry

The Jaime Hewlett / Damon Albarn
that the BBC made for
their 2008 Olympics coverage gives the general idea.

What tools does Wukong work with?

Wukong is friends with Hadoop the elephant,
Pig the query language, and the cat on your
command line. We’re looking forward to being friends with
martinis and express
down the road.