Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
To get started with Scalding, first clone the Scalding repository on Github:
git clone https://github.com/twitter/scalding.git
Next, build the code using sbt (a standard Scala build tool). Make sure you have Scala (download here, see scalaVersion in project/Build.scala for the correct version to download), and run the following commands:
./sbt update ./sbt test # runs the tests; if you do 'sbt assembly' below, these tests, which are long, are repeated ./sbt assembly # creates a fat jar with all dependencies, which is useful when using the scald.rb script
Now you're good to go!
Using Scalding with other versions of Scala
Scalding works with Scala 2.10 and 2.11 is recommended, though a few configuration files must be changed for this to work. In project/Build.scala, ensure that the proper scalaVersion value is set. Additionally, you'll need to ensure the proper version of specs in the same config. Change the following line
libraryDependencies += "org.scala-tools.testing" % "specs_2.10" % "1.6.9" % "test"
You can find the published versions here.
Scala's IDE support is generally not as strong as Java's, but there are several options that some people prefer. Both Eclipse and IntelliJ have plugins that support Scala syntax. To generate a project file for Scalding in Eclipse, refer to this project, and for IntelliJ files, this (note that with the latter, the 1.1 snapshot is recommended).
For a quick introduction into Scalding, design patterns, TDD and connecting with external systems refer to this book Programming MapReduce with Scalding. You can code examples presented in the book here