Adds choice of persistence level to Bagel. #513

merged 3 commits into from Mar 11, 2013


None yet
2 participants

MLnick commented Mar 9, 2013

This PR adds support for selecting the StorageLevel to use when running a Bagel computation. The default remains MEMORY_ONLY and should be fully backward-compatible in terms of run method calls.

Note that due to the way Scala handles default arguments it was not possible to simply add the storageLevel parameter to each run method with a default. Hence I did so for the "main" run method and then added a second method for each of the remaining ones, that takes care of the default argument (and whose signature matches the old versions).

See!topic/spark-developers/F_UOsKM8ZEM for the discussion.

I also added some documentation and a (probably superfluous) test to the BagelSuite to test use of the non-default StorageLevel parameter.

addAggregatorArg[K, V, M, C](compute))
+ /** Runs a Bagel program with no [[spark.bagel.Aggregator]], default [[spark.HashPartitioner]],
+ * [[spark.bagel.DefaultCombiner]] and the default storage level */

mateiz Mar 9, 2013


Looks good, but make the comment formatting consistent with the rest of the codebase; it should look like this:

 * Blah blah

Or, for single-line comments, this is also okay:

/** Blah */

MLnick commented Mar 11, 2013

@mateiz made the suggested doc style changes.

mateiz added a commit that referenced this pull request Mar 11, 2013

Merge pull request #513 from MLnick/bagel-caching
Adds choice of persistence level to Bagel.

@mateiz mateiz merged commit cbf8f0d into mesos:master Mar 11, 2013


mateiz commented Mar 11, 2013

Cool, thanks!

@MLnick MLnick deleted the MLnick:bagel-caching branch Mar 11, 2013

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment