Ruby libraries for efficient, effective Hadoop streaming and bulk data processing
Ruby C Shell
Clone or download
Pull request Compare This branch is 736 commits behind infochimps-labs:master.
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Failed to load latest commit information.



Wukong is Ruby for Hadoop — it makes Hadoop so easy a chimpanzee can use it.

Treat your dataset like a

  • stream of lines when it’s efficient to process by lines
  • stream of field arrays when it’s efficient to deal directly with fields
  • stream of lightweight objects when it’s efficient to deal with objects

Wukong is friends with Hadoop the elephant, Pig the query language, and the cat on your command line.

The main documentation lives on the Wukong Pages. Please feel free to add supplemental information to the wukong wiki.

Imminent Changes

I’m pushing to release “Wukong 3.0 the actual 1.0 release”.

  • For reducing/uniqing, a notion of mutable_fields and immutable_fields and extrinsic_fields: two objects compare the same/differently if their mutable fields compare the same/differently
  • Methods on TypedStruct to
  • Make to_flat(false) the default, with the sort_fields / partition_fields defaulting to 2 each and very prominently documented
  • Standardize the notion that wukong classes have a “key”; by default, it will be to_a.first for Structs/TypedStructs. This shouldn’t break anything.
  • May make some things that are derived classes into mixin’ed modules
  • Will probably change the name of AccumulatingReducer into just Accumulator, and have all Accumulator-derived classes include Accumulator; I’ll make sure the old names continue to work though.


Send Wukong questions to the Infinite Monkeywrench mailing list


Get the code

We’re still actively developing wukong. The newest version is available via Git on github:

$ git clone git://

A gem is available from gemcutter:

$ sudo gem install wukong --source=

(don’t use the version — it’s way out of date.)

You can instead download this project in either zip or tar formats.

Dependencies and setup

To finish setting up, see the detailed setup instructions and then read the usage notes

How to write a Wukong script

Here’s a script to count words in a text stream:

    require 'wukong'
    module WordCount
      class Mapper < Wukong::Streamer::LineStreamer
        # Emit each word in the line.
        def process line
          words = line.strip.split(/\W+/).reject(&:blank?)
          words.each{|word| yield [word, 1] }
      class Reducer < Wukong::Streamer::ListReducer
        def finalize
          yield [ key, ]
      ).run # Execute the script

The first class, the Mapper, eats lines and craps [word, count] records: word is the /key/, its count is the /value/.

In the reducer, the values for each key are stacked up into a list; then the record(s) yielded by #finalize are emitted. There are many other ways to write the reducer (most of them are better) — see the examples

Structured data stream

You can also use structs to treat your dataset as a stream of objects:

    require 'wukong'
    require 'my_blog' #defines the blog models
    # structs for our input objects
    Tweet = :id, :created_at, :twitter_user_id,
      :in_reply_to_user_id, :in_reply_to_status_id, :text )
    TwitterUser  = :id, :username, :fullname,
      :homepage, :location, :description )
    module TwitBlog
      class Mapper < Wukong::Streamer::RecordStreamer
        # Watch for tweets by me
        MY_USER_ID = 24601
        # If this is a tweet is by me, convert it to a Post.
        # If it is a tweet not by me, convert it to a Comment that
        # will be paired with the correct Post.
        # If it is a TwitterUser, convert it to a User record and
        # a user_location record
        def process record
          case record
          when TwitterUser
            user     = # grab the fields in common
            user_loc =, record.location, nil, nil)
            yield user
            yield user_loc
          when Tweet
            if record.twitter_user_id == MY_USER_ID
              post = record
     = "{}"
              post.body = record.text
              post.title = record.text[0..65] + "..."
              yield post
              comment = record
              comment.body    = record.text
              comment.post_id = record.in_reply_to_status_id
              yield comment
    end TwitBlog::Mapper, nil ).run # identity reducer

Advanced Patterns

Wukong has a good collection of map/reduce patterns. Here’s an AccumulatingReducer that takes a long list of key-value pairs and emits, for each key, all its corresponding values in one line.

    # Roll up all values for each key into a single line
    class GroupByReducer < Wukong::Streamer::AccumulatingReducer
      attr_accessor :values

      # Start with an empty list
      def start! *args
        self.values = []

      # Aggregate each value in turn 
      def accumulate key, value
        self.values << value

      # Emit the key and all values, tab-separated
      def finalize
        yield [key, values].flatten

So given adjacency pairs for the following directed friend graph:

    @jerry      @elaine
    @elaine     @jerry
    @jerry      @kramer
    @kramer     @jerry
    @kramer     @bobsacamato
    @kramer     @newman
    @jerry      @superman
    @newman     @kramer
    @newman     @elaine
    @newman     @jerry

You’d end up with

    @elaine     @jerry
    @jerry      @elaine      @kramer     @superman
    @kramer     @bobsacamato @jerry      @newman
    @newman     @elaine      @jerry      @kramer   

Why is it called Wukong?

Hadoop, as you may know, is named after a stuffed elephant. Since Wukong was started by the infochimps team, we needed a simian analog. A Monkey King who journeyed to the land of the Elephant seems to fit the bill:

Sun Wukong (孙悟空), known in the West as the Monkey King, is the main character in the classical Chinese epic novel Journey to the West. In the novel, he accompanies the monk Xuanzang on the journey to retrieve Buddhist sutras from India.

Sun Wukong possesses incredible strength, being able to lift his 13,500 jīn (8,100 kg) Ruyi Jingu Bang with ease. He also has superb speed, traveling 108,000 li (54,000 kilometers) in one somersault. Sun knows 72 transformations, which allows him to transform into various animals and objects; he is, however, shown with slight problems transforming into other people, since he is unable to complete the transformation of his tail. He is a skilled fighter, capable of holding his own against the best generals of heaven. Each of his hairs possesses magical properties, and is capable of transforming into a clone of the Monkey King himself, or various weapons, animals, and other objects. He also knows various spells in order to command wind, part water, conjure protective circles against demons, freeze humans, demons, and gods alike. — Sun Wukong’s Wikipedia entry

The Jaime Hewlett / Damon Albarn short that the BBC made for their 2008 Olympics coverage gives the general idea.

More info

There are many useful examples in the examples/ directory.


Monkeyshines was written by Philip (flip) Kromer ( / @mrflip) for the infochimps project

Patches submitted by:

Thanks to:


Send monkeyshines questions to the Infinite Monkeywrench mailing list