No description, website, or topics provided.
Java PigLatin Python Scala Shell Ruby
Switch branches/tags
Nothing to show
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Failed to load latest commit information.


This repository contains examples (and errata) for Learning Hadoop 2.


Throughout the book we use Cloudera CDH 5.0 and Amazon EMR as reference systems. All examples target, and have been tested with, Java 7.

Build the examples

The easiest way to build the examples, with CDH 5.0 dependencies, is to use the provided Gradle and sbt scripts.


We use Gradle to compile Java code and collect the required class files into a single JAR file.

$ ./gradlew jar

JARs can then be submitted to Hadoop with:

$ hadoop jar <job jarfile> <main class> <argument 1> ... <argument 2>

Example - Chapter 3 (Mapreduce and beyond)

To build ch3 examples

$ git clone
$ cd book-examples/ch3
$ ./gradlew jar

The script will take care of downloading a Gradle distribution from the official repo (, and use it to build the code under src/main/java/com/learninghadoop2/mapreduce/. You will find the resulting jar in build/libs/mapreduce-example.jar.

We can run the WordCount example as described in Chapter 3:

$ hadoop jar build/libs/mapreduce-example.jar \
com.learninghadoop2.mapreduce.WordCount \
input.txt \

For more information on how gradle is bootstrapped to run the build, refer to The gradle_wrapper plugin is distributed with the examples (gradle/wrapper/gradle-wrapper.jar).


We use sbt to build, manage, and execute the Spark examples in Chapter 5.

The build.sbt file controls the codebase metadata and software dependencies.

The source code for all examples can be compiled with:

$ cd ch5
$ sbt compile

Or, it can be packaged into a JAR file with:

$ sbt package

For Spark in standalone mode, an helper script to execute compiled classes can be generated with:

$ sbt add-start-script-tasks
$ sbt start-script

The helper can be invoked as follows:

$ target/start <class name> <master> <param1> … <param n>


To run the examples on a YARN grid on CDH5, you can build a JAR file using:

$ sbt package

and then ship it to the Resource Manager using the spark-submit command:

./bin/spark-submit --class --master yarn-cluster [options] target/scala-2.10/chapter-4_2.10-1.0.jar [<param1> … <param n>]

Unlike the standalone mode, we don't need to specify a URI.

More information on launching Spark on YARN can be found at