Skip to content
A Scala API for Cascading
Scala Java Ruby Shell Thrift
Latest commit 01e2400 May 11, 2016 @johnynek johnynek Merge pull request #1559 from twitter/jnievelt/1558
Removing Option imports for TreeOrderedBuf macros
Failed to load latest commit information.
logo Add scalding logo Oct 25, 2013
maple/src/main/java/com Maple fix for HBaseTap, Issue 781 Mar 2, 2016
project Switch to https based remote repo for DocGen May 3, 2016
scalding-args/src Override hashCode for Args Feb 5, 2015
scalding-avro Move Scalding to 2.10/2.11 Dec 5, 2014
scalding-benchmarks/src/test/scala/com/twitter/scalding import hygiene: remove unused imports Apr 1, 2015
scalding-commons Cleanup cleanup (in VersionedStore) May 4, 2016
scalding-core Prepare for release 0.16.0 May 3, 2016
scalding-date/src Add before() and after() to RichDate Mar 23, 2016
scalding-db clarify comment in jdbc wasNull check May 3, 2016
scalding-hadoop-test/src Skip invoking reducer estimators when reducers are set explicitly Apr 26, 2016
scalding-hraven refactor toFlowStepHistory Jan 14, 2016
scalding-jdbc/src Fixes Dec 5, 2014
scalding-json/src import hygiene: remove unused imports Apr 1, 2015
scalding-parquet-fixtures/src/test/resources Copy over code from parquet-cascading, drop dep Feb 1, 2016
scalding-parquet-scrooge-fixtures/src/test/resources Move the scrooge fixtures into their own build targets so the local b… Aug 3, 2015
scalding-parquet-scrooge Fix pig exclude, unused imports, parquet-scrooge test Feb 1, 2016
scalding-parquet Warning from the parquet-cascading merge in Feb 2, 2016
scalding-repl Use `wartremover` to guard against careless use of `_.get` Feb 8, 2016
scalding-serialization/src Removing Option imports for TreeOrderedBuf macros May 11, 2016
scalding-thrift-macros-fixtures/src/test/resources Because, because... fun, the scala compiler has special naming rules … Mar 3, 2016
scalding-thrift-macros Because, because... fun, the scala compiler has special naming rules … Mar 4, 2016
scripts Fix references to Build.scala in other places Feb 1, 2016
tutorial Prepare for release 0.16.0 May 3, 2016
.gitignore Update the build Feb 1, 2016
.travis.blacklist Add scalding-parquet-fixtures to travis blacklist Feb 1, 2016
.travis.yml Update Scala version to 2.10.6 Dec 27, 2015 Remove two extra changes May 3, 2016 Add Sep 19, 2012
LICENSE Initial Import Jan 10, 2012
NOTICE Initial Import Jan 10, 2012 Prepare for release 0.16.0 May 3, 2016
build.sbt Fix `wartremover` to only run during the `compile` phase Feb 10, 2016
sbt Update Scala version to 2.10.6 Dec 27, 2015
version.sbt Set new version to 0.16.1-SNAPSHOT May 3, 2016


Scalding is a Scala library that makes it easy to specify Hadoop MapReduce jobs. Scalding is built on top of Cascading, a Java library that abstracts away low-level Hadoop details. Scalding is comparable to Pig, but offers tight integration with Scala, bringing advantages of Scala to your MapReduce jobs.

Scalding Logo

Current version: 0.16.0

Word Count

Hadoop is a distributed system for counting words. Here is how it's done in Scalding.

package com.twitter.scalding.examples

import com.twitter.scalding._
import com.twitter.scalding.source.TypedText

class WordCountJob(args: Args) extends Job(args) {
    .flatMap { line => tokenize(line) }
    .groupBy { word => word } // use each word for a key
    .size // in each group, get the size
    .write(TypedText.tsv[(String, Long)](args("output")))

  // Split a piece of text into individual words.
  def tokenize(text: String): Array[String] = {
    // Lowercase each word and remove punctuation.
    text.toLowerCase.replaceAll("[^a-zA-Z0-9\\s]", "").split("\\s+")

Notice that the tokenize function, which is standard Scala, integrates naturally with the rest of the MapReduce job. This is a very powerful feature of Scalding. (Compare it to the use of UDFs in Pig.)

You can find more example code under examples/. If you're interested in comparing Scalding to other languages, see our Rosetta Code page, which has several MapReduce tasks in Scalding and other frameworks (e.g., Pig and Hadoop Streaming).

Documentation and Getting Started

Please feel free to use the beautiful Scalding logo artwork anywhere.

Code of Conduct

This, and all projects, are under the Twitter Open Source Code of Conduct. Additionally, see the Typelevel Code of Conduct for specific examples of harassing behavior that are not tolerated.


There is a script (called sbt) in the root that loads the correct sbt version to build:

  1. ./sbt update (takes 2 minutes or more)
  2. ./sbt test
  3. ./sbt assembly (needed to make the jar used by the scald.rb script)

The test suite takes a while to run. When you're in sbt, here's a shortcut to run just one test:

> test-only com.twitter.scalding.FileSourceTest

Please refer to FAQ page if you encounter problems when using sbt.

We use Travis CI to verify the build: Build Status

We use Coveralls for code coverage results: Coverage Status

Scalding modules are available from maven central.

The current groupid and version for all modules is, respectively, "com.twitter" and 0.16.0-RC1.

Current published artifacts are

  • scalding-core_2.10
  • scalding-args_2.10
  • scalding-date_2.10
  • scalding-commons_2.10
  • scalding-avro_2.10
  • scalding-parquet_2.10
  • scalding-repl_2.10

The suffix denotes the scala version.


  • Ebay
  • Etsy
  • Sharethrough
  • Snowplow Analytics
  • Soundcloud
  • Twitter

To see a full list of users or to add yourself, see the wiki


For user questions or scalding development (internals, extending, release planning):!forum/scalding-dev (Google search also works as a first step)

In the remote possibility that there exist bugs in this code, please report them to:

Follow @Scalding on Twitter for updates.

Chat: Gitter


Thanks for assistance and contributions:

A full list of contributors can be found on GitHub.


Copyright 2013 Twitter, Inc.

Licensed under the Apache License, Version 2.0:

Something went wrong with that request. Please try again.