Skip to content


Subversion checkout URL

You can clone with
Download ZIP
A Scala API for Cascading
Scala Java Ruby Shell Thrift

Merge pull request #1444 from tdyas/optin_for_jobtest_validate

opt-in to calling Job.validate in JobTest
latest commit 398545991c
@ianoc ianoc authored
Failed to load latest commit information.
logo Add scalding logo
maple/src/main/java/com remove unused imports from maple java code
project Updated chill version
scalding-args/src Override hashCode for Args
scalding-avro Move Scalding to 2.10/2.11
scalding-benchmarks/src/test/scala/com/twitter/scalding import hygiene: remove unused imports
scalding-commons Fixes map-only exception when setting both source and sink binary con…
scalding-core opt-in to calling Job.validate in JobTest
scalding-date/src Fixes docs wording
scalding-db Collapses scalding-db packages
scalding-hadoop-test/src make getBytesPerReducer support human readable values like 128m and 1g
scalding-hraven Update RuntimeReducerEstimator
scalding-jdbc/src Fixes
scalding-json/src import hygiene: remove unused imports
scalding-parquet-scrooge-fixtures/src/test/resources Move the scrooge fixtures into their own build targets so the local b…
scalding-parquet-scrooge Add the type in ScroogeReadSupport
scalding-parquet Use a different fix for Parquet346
scalding-repl Remove publishArtifact, reverse scalding_repl list
scalding-serialization/src Serialization folding
scalding-thrift-macros-fixtures/src/test/resources Move the scrooge fixtures into their own build targets so the local b…
scalding-thrift-macros Move the scrooge fixtures into their own build targets so the local b…
scripts Remove the bootstrap section
tutorial Change ScaldingILoop
.gitignore minor clean up
.travis.blacklist Add travis blacklist entries for fixtures
.travis.yml Update .travis.yml bumping version to 0.15.0; algebird to 0.10.1 Add
LICENSE Initial Import
NOTICE Initial Import Fix Readme travis link
sbt Upgrade sbt launcher script (sbt-extras)
version.sbt fixed version


Scalding is a Scala library that makes it easy to specify Hadoop MapReduce jobs. Scalding is built on top of Cascading, a Java library that abstracts away low-level Hadoop details. Scalding is comparable to Pig, but offers tight integration with Scala, bringing advantages of Scala to your MapReduce jobs.

Scalding Logo

Current version: 0.15.0

Word Count

Hadoop is a distributed system for counting words. Here is how it's done in Scalding.

package com.twitter.scalding.examples

import com.twitter.scalding._

class WordCountJob(args: Args) extends Job(args) {
    .flatMap { line => tokenize(line) }
    .groupBy { word => word } // use each word for a key
    .size // in each group, get the size
    .write(TypedTsv[(String, Long)](args("output")))

  // Split a piece of text into individual words.
  def tokenize(text: String): Array[String] = {
    // Lowercase each word and remove punctuation.
    text.toLowerCase.replaceAll("[^a-zA-Z0-9\\s]", "").split("\\s+")

Notice that the tokenize function, which is standard Scala, integrates naturally with the rest of the MapReduce job. This is a very powerful feature of Scalding. (Compare it to the use of UDFs in Pig.)

You can find more example code under examples/. If you're interested in comparing Scalding to other languages, see our Rosetta Code page, which has several MapReduce tasks in Scalding and other frameworks (e.g., Pig and Hadoop Streaming).

Documentation and Getting Started

Please feel free to use the beautiful Scalding logo artwork anywhere.

Code of Conduct

This, and all projects, are under the Twitter Open Source Code of Conduct. Additionally, see the Typelevel Code of Conduct for specific examples of harassing behavior that are not tolerated.


There is a script (called sbt) in the root that loads the correct sbt version to build:

  1. ./sbt update (takes 2 minutes or more)
  2. ./sbt test
  3. ./sbt assembly (needed to make the jar used by the scald.rb script)

The test suite takes a while to run. When you're in sbt, here's a shortcut to run just one test:

> test-only com.twitter.scalding.FileSourceTest

Please refer to FAQ page if you encounter problems when using sbt.

We use Travis CI to verify the build: Build Status

We use Coveralls for code coverage results: Coverage Status

Scalding modules are available from maven central.

The current groupid and version for all modules is, respectively, "com.twitter" and 0.12.0.

Current published artifacts are

  • scalding-core_2.10
  • scalding-args_2.10
  • scalding-date_2.10
  • scalding-commons_2.10
  • scalding-avro_2.10
  • scalding-parquet_2.10
  • scalding-repl_2.10

The suffix denotes the scala version.


  • Ebay
  • Etsy
  • Sharethrough
  • Snowplow Analytics
  • Soundcloud
  • Twitter

To see a full list of users or to add yourself, see the wiki


For user questions, we are using the cascading-user mailing list for discussions:

For scalding development (internals, extending, release planning):!forum/scalding-dev

In the remote possibility that there exist bugs in this code, please report them to:

Follow @Scalding on Twitter for updates.

Chat (IRC): freenode channel: #scalding


Thanks for assistance and contributions:

A full list of contributors can be found on GitHub.


Copyright 2013 Twitter, Inc.

Licensed under the Apache License, Version 2.0:

Something went wrong with that request. Please try again.