A Scala API for Cascading
Switch branches/tags
0.16.0 0.17.x alexlevenson/empty-success alexlevenson/parquet-1.8.1 alexlevenson/sbt-launcher-upgrade alexlevenson/typed-pipe-diff-fix alexlevenson/1541 bazel-build benpence-patch-1 benpence-patch-2 cascading3 dcl/exec_fun dcl/exec_listener dependent-mode develop exie/add_fetch_size exie/1068 exie/1135_try_another_branch fbs-staging-macros-scrooge-tbase fbs-staging-macros-scrooge fix_to_iterable_execution ggonzalez/fewer_gets ggonzalez/fix_readPathsFor ggonzalez/safer_box ggonzalez/0.15.1-rc1 gh-pages ianoc/MoveDefaultBuildsTo211 ianoc/addSketchOrderedSerializationInterface ianoc/genericSource ianoc/housekeeping ianoc/jdbcMacros ianoc/lui ianoc/removeUnneededTypeParameter ianoc/scaldingViz ioconnell/SeparateFS ioconnell/addMacroDescriptions ioconnell/sparkScalding isnotinvain-patch-1 isnotinvain-patch-2-1 isnotinvain-patch-2 mashraf/scalding_RC15 mashraf/scalding_rc16 mashraf/sequential_build mashraf/0.15.1-rc1 master oscar/add-a-regression-test oscar/add-deprecated-mode-methods oscar/add-flow-count oscar/add-require-singlevalue oscar/add_memory_backend_regression_1802 oscar/add_spark_sources_sinks oscar/better-reduce-adt oscar/bump-dagon oscar/bump-scala-versions oscar/check-fan-out oscar/chunking oscar/cogroup-adt oscar/execution-caching-fix oscar/execution_cleanup_on_finish oscar/fix-build_sbt_deps oscar/fix-config-equals-hashcode oscar_fix_bdd-typedtext oscar/fix_pair_joinf_closure oscar/good_tuple_conversion_equality oscar/hellcats oscar/improve-memory-platform oscar/improve_memory_backend oscar/improve_write_partitions oscar/issue_1786 oscar/makeAddTrapDeprecated oscar/memoize-hashCode-equals-typedpipe oscar/merge-develop-cascading3-more-planning-tests oscar/merge-develop-cascading3 oscar/more-ast1 oscar/more-ast3 oscar/more-ast4 oscar/more-ast5 oscar/more-hashjoin-rules oscar/more-warts oscar/mv-summingbird-graph-lib oscar/no-literals oscar/no-mode-tpinst oscar/optimize-sum-joins oscar/pipe_ser oscar/properly-memoize-tp-equals oscar/pure-counters oscar/refactor_evalcache oscar/remove-graph-lib oscar/remove-some-field-implicits oscar/remove_n2_check_cascading3 oscar/repro-toiterableexecution-issue_017 oscar/robust-serializer oscar/scalafmt oscar/simplify_memory_planner_with_dagon oscar/spark_improvements_2018_08_19_spark_writer oscar/spark_session_instead_of_context oscar/spark oscar/syntax_for_sketch_join oscar/test_typedsource_toiterable oscar/typedpipe-ast oscar/use-batched-in-sketched oscar/use-dagon-2 oscar/use-dagon-3 oscar/use-dagon oscar/use-scalafmt-0_5 oscar/weaken-the-cache oscar/0.16.0-RC1 oscar/0.16.0-RC2 oscar/0_17_4_cherry-picks pg/fix_typed_pipe_monoid piyush/cascading3 piyush/memory_estimation piyush/tez_partitions_config releases/0.15 revertUnitOrderedSerialization rubanm/cascading3/distcachetap_support rubanm/cascading3/snapshot rubanm/cascading3/tez_partitions_config rubanm/move_cascading_schemes_to_subprojects rubanm/parquet_cascading3 rubanm/v0.16.1-cascading3-RC2 rubanm/0.16.1-RC1_typed_mapside_reduce_revert rubanm/0.16.1-RC1 rubanm/0.16.1-RC3 sbt_0_13_7 scoverage simple-source sritchie/jvm_opts sritchie/upgrade212 twitter_release twitter_0.17.2_with_fix_filesource typed-transient typed_jdbc_source unitOrderedSerialization update_dagon_0_2_4
Nothing to show
Clone or download
johnynek Add scalding-cats module for cats typeclasses on scalding types (#1890)
* Add scalding-cats module for cats typeclasses on scalding types

* add tests for Effect, Async, MonadError, etc.

* minor cleanup

* simplify and add a comment

* fix to pass the tests with stronger gens

* actually run the tests in CI
Latest commit fcd3d1b Dec 10, 2018
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
docs/src/main Generate Scalding microsite via sbt-microsites (#1623) Nov 19, 2016
logo Add scalding logo Oct 25, 2013
maple/src/main/java/com Upgrade hbase version up to 1.2.4 Jul 20, 2018
project Remove scalafix because broken scalading-quotation tests Jul 30, 2018
scalding-args/src Use reflection over Jobs to find serialized classes (#1654) Apr 7, 2017
scalding-avro add explicit return types to implicit methods and values (#1660) Apr 5, 2017
scalding-benchmarks/src/test/scala/com/twitter/scalding import hygiene: remove unused imports Apr 1, 2015
scalding-cats/src Add scalding-cats module for cats typeclasses on scalding types (#1890) Dec 10, 2018
scalding-commons The problem with combination TupleEntrySchemeIterator and "bad" recor… Dec 4, 2018
scalding-core Add scalding-cats module for cats typeclasses on scalding types (#1890) Dec 10, 2018
scalding-date/src Remove unused imports Jul 30, 2018
scalding-db Remove unused imports Jul 30, 2018
scalding-estimators-test/src/test Remove unused imports Jul 30, 2018
scalding-hadoop-test/src Remove unused imports Jul 30, 2018
scalding-hraven remove a few warnings (#1720) Sep 24, 2017
scalding-jdbc/src remove a few warnings (#1720) Sep 24, 2017
scalding-json/src Make scalding native MemoryBackend usable (#1782) Feb 6, 2018
scalding-parquet-fixtures/src/test/resources Add partitioned parquet thrift source Aug 30, 2016
scalding-parquet-scrooge-fixtures/src/test/resources Scalding updates for Scala 2.12 (#1646) Mar 23, 2017
scalding-parquet-scrooge Allowing backwards compatible projections (#1882) Dec 4, 2018
scalding-parquet Turn on more warts (#1723) Sep 24, 2017
scalding-quotation/src Remove scalafix because broken scalading-quotation tests Jul 30, 2018
scalding-repl Remove unused imports Jul 30, 2018
scalding-serialization/src Remove unused imports Jul 30, 2018
scalding-spark/src Use SparkSession instead of SparkContext (#1870) Aug 22, 2018
scalding-thrift-macros-fixtures/src/test/resources Because, because... fun, the scala compiler has special naming rules … Mar 4, 2016
scalding-thrift-macros Spark improvements (#1868) Aug 21, 2018
scripts Cherry pick 0.17.2 release related changes into develop (#1710) Jul 15, 2017
tutorial remove a few warnings (#1720) Sep 24, 2017
.gitignore introduce scalding-quotation sub-project (#1755) Dec 18, 2017
.travis.blacklist Add scalding-parquet-fixtures to travis blacklist Feb 1, 2016
.travis.yml Add scalding-cats module for cats typeclasses on scalding types (#1890) Dec 10, 2018
CHANGES.md Cherry pick 0.17.2 release related changes into develop (#1710) Jul 15, 2017
COMMITTERS.md Create COMMITTERS.md Aug 26, 2016
CONTRIBUTING.md Fixed Spelling. Aug 10, 2018
LICENSE Initial Import Jan 10, 2012
NOTICE Initial Import Jan 10, 2012
README.md Cherry pick 0.17.2 release related changes into develop (#1710) Jul 15, 2017
build.sbt Add scalding-cats module for cats typeclasses on scalding types (#1890) Dec 10, 2018
sbt Update the sbt runner script May 22, 2017
version.sbt Cherry pick 0.17.2 release related changes into develop (#1710) Jul 15, 2017

README.md

Scalding

Build status Coverage Status Latest version Chat

Scalding is a Scala library that makes it easy to specify Hadoop MapReduce jobs. Scalding is built on top of Cascading, a Java library that abstracts away low-level Hadoop details. Scalding is comparable to Pig, but offers tight integration with Scala, bringing advantages of Scala to your MapReduce jobs.

Scalding Logo

Word Count

Hadoop is a distributed system for counting words. Here is how it's done in Scalding.

package com.twitter.scalding.examples

import com.twitter.scalding._
import com.twitter.scalding.source.TypedText

class WordCountJob(args: Args) extends Job(args) {
  TypedPipe.from(TextLine(args("input")))
    .flatMap { line => tokenize(line) }
    .groupBy { word => word } // use each word for a key
    .size // in each group, get the size
    .write(TypedText.tsv[(String, Long)](args("output")))

  // Split a piece of text into individual words.
  def tokenize(text: String): Array[String] = {
    // Lowercase each word and remove punctuation.
    text.toLowerCase.replaceAll("[^a-zA-Z0-9\\s]", "").split("\\s+")
  }
}

Notice that the tokenize function, which is standard Scala, integrates naturally with the rest of the MapReduce job. This is a very powerful feature of Scalding. (Compare it to the use of UDFs in Pig.)

You can find more example code under examples/. If you're interested in comparing Scalding to other languages, see our Rosetta Code page, which has several MapReduce tasks in Scalding and other frameworks (e.g., Pig and Hadoop Streaming).

Documentation and Getting Started

Please feel free to use the beautiful Scalding logo artwork anywhere.

Contact

For user questions or scalding development (internals, extending, release planning): https://groups.google.com/forum/#!forum/scalding-dev (Google search also works as a first step)

In the remote possibility that there exist bugs in this code, please report them to: https://github.com/twitter/scalding/issues

Follow @Scalding on Twitter for updates.

Chat: Gitter

Get Involved + Code of Conduct

Pull requests and bug reports are always welcome!

We use a lightweight form of project governence inspired by the one used by Apache projects. Please see Contributing and Committership for our code of conduct and our pull request review process. The TL;DR is send us a pull request, iterate on the feedback + discussion, and get a +1 from a Committer in order to get your PR accepted.

The current list of active committers (who can +1 a pull request) can be found here: Committers

A list of contributors to the project can be found here: Contributors

Building

There is a script (called sbt) in the root that loads the correct sbt version to build:

  1. ./sbt update (takes 2 minutes or more)
  2. ./sbt test
  3. ./sbt assembly (needed to make the jar used by the scald.rb script)

The test suite takes a while to run. When you're in sbt, here's a shortcut to run just one test:

> test-only com.twitter.scalding.FileSourceTest

Please refer to FAQ page if you encounter problems when using sbt.

We use Travis CI to verify the build: Build Status

We use Coveralls for code coverage results: Coverage Status

Scalding modules are available from maven central.

The current groupid and version for all modules is, respectively, "com.twitter" and 0.17.2.

Current published artifacts are

  • scalding-core_2.11, scalding-core_2.12
  • scalding-args_2.11, scalding-args_2.12
  • scalding-date_2.11, scalding-date_2.12
  • scalding-commons_2.11, scalding-commons_2.12
  • scalding-avro_2.11, scalding-avro_2.12
  • scalding-parquet_2.11, scalding-parquet_2.12
  • scalding-repl_2.11, scalding-repl_2.12

The suffix denotes the scala version.

Adopters

  • Ebay
  • Etsy
  • Sharethrough
  • Snowplow Analytics
  • Soundcloud
  • Twitter

To see a full list of users or to add yourself, see the wiki

Authors:

Thanks for assistance and contributions:

A full list of contributors can be found on GitHub.

License

Copyright 2016 Twitter, Inc.

Licensed under the Apache License, Version 2.0