Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP
A Scala API for Cascading
Scala Ruby Java Shell

Fetching latest commit…

Cannot retrieve the latest commit at this time

Failed to load latest commit information.
project
scalding-args/src
scalding-core
scalding-date/src
scripts
tutorial
.gitignore
.travis.yml
CHANGES.md
CONTRIBUTING.md
LICENSE
NOTICE
README.md
version.sbt

README.md

Scalding

Scalding is a Scala library that makes it easy to specify Hadoop MapReduce jobs. Scalding is built on top of Cascading, a Java library that abstracts away low-level Hadoop details. Scalding is comparable to Pig, but offers tight intergation with Scala, bringing advantages of Scala to your MapReduce jobs.

Current version: 0.8.11

Word Count

Hadoop is a distributed system for counting words. Here is how it's done in Scalding.

package com.twitter.scalding.examples

import com.twitter.scalding._

class WordCountJob(args : Args) extends Job(args) {
  TextLine( args("input") )
    .flatMap('line -> 'word) { line : String => tokenize(line) }
    .groupBy('word) { _.size }
    .write( Tsv( args("output") ) )

  // Split a piece of text into individual words.
  def tokenize(text : String) : Array[String] = {
    // Lowercase each word and remove punctuation.
    text.toLowerCase.replaceAll("[^a-zA-Z0-9\\s]", "").split("\\s+")
  }
}

Notice that the tokenize function, which is standard Scala, integrates naturally with the rest of the MapReduce job. This is a very powerful feature of Scalding. (Compare it to the use of UDFs in Pig.)

You can find more example code under examples/. If you're interested in comparing Scalding to other languages, see our Rosetta Code page, which has several MapReduce tasks in Scalding and other frameworks (e.g., Pig and Hadoop Streaming).

Documentation and Getting Started

Building

  1. Install sbt 0.12.2 (sorry, but the assembly plugin is sbt version dependent).
  2. sbt update (takes 2 minutes or more)
  3. sbt test
  4. sbt assembly (needed to make the jar used by the scald.rb script)

The test suite takes a while to run. When you're in sbt, here's a shortcut to run just one test:

> test-only com.twitter.scalding.FileSourceTest

Please refer to FAQ page if you encounter problems when using sbt.

We use Travis CI to verify the build: Build Status

The current version is 0.8.11 and is available from maven central: org="com.twitter", artifact="scalding-core_2.9.3".

Contact

Currently we are using the cascading-user mailing list for discussions: http://groups.google.com/group/cascading-user

In the remote possibility that there exist bugs in this code, please report them to: https://github.com/twitter/scalding/issues

Follow @Scalding on Twitter for updates.

Authors:

Thanks for assistance and contributions:

License

Copyright 2013 Twitter, Inc.

Licensed under the Apache License, Version 2.0: http://www.apache.org/licenses/LICENSE-2.0

Something went wrong with that request. Please try again.