Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP
Common extensions to the Scalding MapReduce DSL.
Scala Perl
Pull request Compare This branch is 47 commits ahead of simonandluna:develop.

Fetching latest commit…

Cannot retrieve the latest commit at this time

Failed to load latest commit information.
src
.gitignore
.travis.yml
CHANGES.md
LICENSE
NOTICE
README.md
build.sbt

README.md

scalding-commons Build Status

Common extensions to the Scalding MapReduce DSL.

Dfs-Datastores Integration

Scalding-Commons includes Scalding Sources for use with the dfs-datastores project.

This library provides a VersionedKeyValSource that allows Scalding to write out key-value pairs of any type into a binary sequencefile. Serialization is handled with the bijection-core library's Bijection trait.

VersionedKeyValSource allows multiple writes to the same path,as write creates a new version. Optionally, given a Monoid on the value type, VersionedKeyValSource allows for versioned incremental updates of a key-value database.

import com.twitter.scalding.source.VersionedKeyValSource
import VersionedKeyValSource._

// ## Sink Example

// The bijection library provides implicit Bijections
// from String -> Array[Byte] and Int -> Array[Byte].
val versionedSource = VersionedKeyValSource[String,Int]("path")

// creates a new version on each write
someScaldingFlow.write(versionedSource)

// because Scalding provides an implicit Monoid[Int],
// the writeIncremental method will add new integers into
// each value on every write:
someScaldingFlow.writeIncremental(versionedSource)

// ## Source Examples
//
// This Source produces the most recent set of kv pairs from the VersionedStore
// located at "path":
VersionedKeyValSource[String,Int]("path")

// This source produces version 12345:
VersionedKeyValSource[String,Int]("path", Some(12345))

Maven

Current version is 0.1.1. groupid="com.twitter", artifact=scalding-commons_2.9.2".

Authors

License

Copyright 2012 Twitter, Inc.

Licensed under the Apache License, Version 2.0: http://www.apache.org/licenses/LICENSE-2.0

Something went wrong with that request. Please try again.