Skip to content
Go to file

Latest commit


Git stats


Failed to load latest commit information.
Latest commit message
Commit time


Hadoop Cluster Management with Intelligent Defaults.


Hadoop is an Apache java framework that allows for distributed processing of enormous datasets across large clusters. It combines a computation engine based on MapReduce with HDFS, a distributed filesystem based on the Google File System.

Abstraction layers such as Cascading (for Java) and Cascalog (for Clojure) make writing MapReduce queries quite nice. Indeed, running hadoop locally with cascalog couldn't be easier.

Unfortunately, graduating one's MapReduce jobs to the cluster level isn't so easy. Amazon's Elastic MapReduce is a great option for getting up and running fast; but what to do if you want to configure your own cluster?

After surveying existing tools, I decided to write my own layer over Pallet, a wonderful cloud provisioning library written in Clojure. Pallet runs on top of jclouds, which allows pallet to define its operations independent of any one cloud provider. Switching between clouds involves a change of login credentials, nothing more.

Getting Started

To include pallet-hadoop in your project, add the following lines to :dev-dependencies in your project.clj file:

[com.palletops/pallet-hadoop "0.3.5"]
[com.palletops/pallet-jclouds "1.7.1"]
[org.apache.jclouds/jclouds-all "1.7.1"]
[org.apache.jclouds.driver/jclouds-sshj "1.7.1"]
[org.apache.jclouds.driver/jclouds-slf4j "1.7.1"]
[ch.qos.logback/logback-classic "1.0.1"]
[ch.qos.logback/logback-core "1.0.1"]

You'll also need to add the Sonatype repository, to get access to Pallet. Add this k-v pair to your project.clj file:

:repositories {"sonatype" ""}

For a detailed example of how to run Pallet-Hadoop, see the example project here. For more detailed information on the project's design, see the project wiki.

Pallet-Hadoop version 0.3.4-SNAPSHOT uses Pallet 0.7.5, jclouds 1.7.1 and Clojure 1.4 and later.


Hadoop Cluster Management with Intelligent Defaults



No packages published
You can’t perform that action at this time.