index.textile

layout	title	collapse
default	mrflip.github.com/wukong	true

wukong :: hadoop made easy

{{ site.description }}

Treat your dataset like a

• stream of lines when it’s efficient to process by lines
• stream of field arrays when it’s efficient to deal directly with fields
• stream of lightweight objects when it’s efficient to deal with objects

Wukong is friends with Hadoop the elephant, Pig the query language, and the cat on your command line.

Send Wukong questions to the Infinite Monkeywrench mailing list

Documentation index

• Install and set up wukong
  • Get the code
  • Setup
  • Installing and Running Wukong with Hadoop
  • Installing and Running Wukong with Datamapper, ActiveRecord, the command-line and more

• Tutorial
  • Count Words
  • Structured data
  • Accumulators including a UniqByLastReducer and a GroupBy reducer.

• Usage notes
  • How to run a Wukong script
  • How to test your scripts
  • Wukong Plays nicely with others
  • Schema export to Pig and SQL
  • Using wukong with internal streaming
  • Using wukong to Batch-Process ActiveRecord Objects

• Wutils — command-line utilies for working with data from the command line
  • Overview of wutils — command listing
  • Stupid command-line tricks using the wutils
  • wu-lign — present a tab-separated file as aligned columns
  • Dear Lazyweb, please build us a tab-oriented version of the Textutils library

• Links and tips for configuring and working with hadoop
• Some opinionated thoughts on working with big data, on why you should drop acid, treat exceptions as records, and happily embrace variable-length strings as primary keys.
• Wukong is licensed under the Apache License (same as Hadoop)

• More info
  • Why is it called Wukong?
  • Don’t Use Wukong, use this instead
  • Further Reading and useful links
  • Note on Patches/Pull Requests
  • What’s up with Wukong::AndPig?
  • Map/Reduce Algorithms
  • TODOs

• Work in progress: an intro to data processing with wukong:
  • Part 1, Get Ready
  • Part 2, Thinking Big Data
  • Part 3, Parsing

{% include intro.textile %}

Credits

Wukong was written by Philip (flip) Kromer (flip@infochimps.org / @mrflip) for the infochimps project

Patches submitted by:

• gemified by Ben Woosley (ben.woosley@gmail.com)
• ruby interpreter path fix by Yuichiro MASUI – masui@masuidrive.jp – http://blog.masuidrive.jp/

Thanks to:

• Brad Heintz for his early feedback
• Phil Ripperger for his wukong in the Amazon AWS cloud tutorial.

Help!

Send Wukong questions to the Infinite Monkeywrench mailing list

{% include news.html %}