Getting Started

Installation

To get started with Scalding, first clone the Scalding repository on Github:

git clone https://github.com/twitter/scalding.git

Next, build the code using sbt (a standard Scala build tool). Make sure you have Scala (download here, see scalaVersion in project/Build.scala for the correct version to download), and run the following commands:

./sbt update
./sbt test     # runs the tests; if you do 'sbt assembly' below, these tests, which are long, are repeated
./sbt assembly # creates a fat jar with all dependencies, which is useful when using the scald.rb script

Now you're good to go!

Using Scalding with other versions of Scala

Scalding works with Scala 2.10 and 2.11 is recommended, though a few configuration files must be changed for this to work. In project/Build.scala, ensure that the proper scalaVersion value is set. Additionally, you'll need to ensure the proper version of specs in the same config. Change the following line

libraryDependencies += "org.scala-tools.testing" % "specs_2.10" % "1.6.9" % "test"

You can find the published versions here.

IDE Support

Scala's IDE support is generally not as strong as Java's, but there are several options that some people prefer. Both Eclipse and IntelliJ have plugins that support Scala syntax. To generate a project file for Scalding in Eclipse, refer to this project, and for IntelliJ files, this (note that with the latter, the 1.1 snapshot is recommended).

Reading material

For a quick introduction into Scalding, design patterns, TDD and connecting with external systems refer to this book Programming MapReduce with Scalding. You can code examples presented in the book here