Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP
Hadoop Cluster Management with Intelligent Defaults
Clojure
Failed to load latest commit information.
dev moved logging file over to dev dependencies, so as not to interfere w…
src/pallet_hadoop Merge pull request #12 from solackerman/my-fix
test/pallet_hadoop updates to project before pallet 0.4.0 release.
.gitignore updated version number, and added missing quote.
.travis.yml prepeare for lein-pallet-release
PLANS.md Moved all future plans to PLANS. Set stage for README updates.
README.md
ReleaseNotes.md
profiles.clj prepeare for lein-pallet-release
project.clj Updated project.clj, release notes and readme for 0.3.4

README.md

Pallet-Hadoop

Hadoop Cluster Management with Intelligent Defaults.

Background

Hadoop is an Apache java framework that allows for distributed processing of enormous datasets across large clusters. It combines a computation engine based on MapReduce with HDFS, a distributed filesystem based on the Google File System.

Abstraction layers such as Cascading (for Java) and Cascalog (for Clojure) make writing MapReduce queries quite nice. Indeed, running hadoop locally with cascalog couldn't be easier.

Unfortunately, graduating one's MapReduce jobs to the cluster level isn't so easy. Amazon's Elastic MapReduce is a great option for getting up and running fast; but what to do if you want to configure your own cluster?

After surveying existing tools, I decided to write my own layer over Pallet, a wonderful cloud provisioning library written in Clojure. Pallet runs on top of jclouds, which allows pallet to define its operations independent of any one cloud provider. Switching between clouds involves a change of login credentials, nothing more.

Getting Started

To include pallet-hadoop in your project, add the following lines to :dev-dependencies in your project.clj file:

[com.palletops/pallet-hadoop "0.3.5"]
[com.palletops/pallet-jclouds "1.7.1"]
[org.apache.jclouds/jclouds-all "1.7.1"]
[org.apache.jclouds.driver/jclouds-sshj "1.7.1"]
[org.apache.jclouds.driver/jclouds-slf4j "1.7.1"]
[ch.qos.logback/logback-classic "1.0.1"]
[ch.qos.logback/logback-core "1.0.1"]

You'll also need to add the Sonatype repository, to get access to Pallet. Add this k-v pair to your project.clj file:

:repositories {"sonatype" "http://oss.sonatype.org/content/repositories/releases/"}

For a detailed example of how to run Pallet-Hadoop, see the example project here. For more detailed information on the project's design, see the project wiki.

Pallet-Hadoop version 0.3.4-SNAPSHOT uses Pallet 0.7.5, jclouds 1.7.1 and Clojure 1.4 and later.

Something went wrong with that request. Please try again.