Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

Update README #6

Merged
merged 1 commit into from

2 participants

@espringe

Scoobi has a combine primitive, which works just like your reduce (using hadoop's combine).

And scoobi also has really great support for transparent serialization. There's a bit of information here: https://github.com/NICTA/scoobi/wiki/Serialization (Note: the explicit stuff is very rarely required, it's just used to allow you to provide your own efficient schemes). And the pretty robust object graph serialization happens here: https://github.com/NICTA/scoobi/blob/master/src/main/scala/com/nicta/scoobi/impl/rtt/ClassBuilder.scala (It's actually using reflection to turn object graph into Java code that would completely reproduce it, then compiles it, and sends the byte code down)

@espringe

Original comment:
espringe/scalding@6efa0ea#commitcomment-942149

(For some reason, github only attaches the comments to the commit, rather than to the pull request)

@johnynek thanks, I've updated the commit

@espringe

@johnynek Uggh, sorry -- I'm an idiot, I hadn't noticed I spelt that wrong too. Fixed

@johnynek johnynek merged commit 2f2bed5 into from
@johnynek
Collaborator

Thanks! Merged.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Commits on Feb 9, 2012
  1. @espringe
This page is out of date. Refresh to see the latest.
Showing with 3 additions and 3 deletions.
  1. +3 −3 README.md
View
6 README.md
@@ -60,10 +60,10 @@ and the win is not so clear.
Scalding supports using any scala object in your map/reduce operations using Kryo serialization,
including scala Lists, Sets,
-Maps, Tuples, etc. It is not clear that such transparent serialization is present yet in either scoobi or
+Maps, Tuples, etc. It is not clear that such transparent serialization is present yet in
scrunch. Like Scoobi, Scalding has a form of MSCR fusion by relying on Cascading's AggregateBy
-operations. Our Reduce primitives (see GroupBuilder.reduce and .mapReduceMap)
-by default uses Hadoop combiners on the map side.
+operations. Our Reduce primitives (see GroupBuilder.reduce and .mapReduceMap) are comparable to
+Scoobi's combine primitive, which by default uses Hadoop combiners on the map side.
Lastly, Scalding comes with a script that allows you to write a single file and run that
single file locally or on your Hadoop cluster by typing one line "scald.rb [--local] myJob.scala".
Something went wrong with that request. Please try again.