Adds choice of persistence level to Bagel. #513

Merged
merged 3 commits into from Mar 11, 2013

Conversation

Projects
None yet
2 participants
Contributor

MLnick commented Mar 9, 2013

This PR adds support for selecting the StorageLevel to use when running a Bagel computation. The default remains MEMORY_ONLY and should be fully backward-compatible in terms of run method calls.

Note that due to the way Scala handles default arguments it was not possible to simply add the storageLevel parameter to each run method with a default. Hence I did so for the "main" run method and then added a second method for each of the remaining ones, that takes care of the default argument (and whose signature matches the old versions).

See https://groups.google.com/forum/?fromgroups=#!topic/spark-developers/F_UOsKM8ZEM for the discussion.

I also added some documentation and a (probably superfluous) test to the BagelSuite to test use of the non-default StorageLevel parameter.

@mateiz mateiz commented on an outdated diff Mar 9, 2013

bagel/src/main/scala/spark/bagel/Bagel.scala
addAggregatorArg[K, V, M, C](compute))
}
+ /** Runs a Bagel program with no [[spark.bagel.Aggregator]], default [[spark.HashPartitioner]],
+ * [[spark.bagel.DefaultCombiner]] and the default storage level */
@mateiz

mateiz Mar 9, 2013

Owner

Looks good, but make the comment formatting consistent with the rest of the codebase; it should look like this:

/**
 * Blah blah
 */

Or, for single-line comments, this is also okay:

/** Blah */
Contributor

MLnick commented Mar 11, 2013

@mateiz made the suggested doc style changes.

@mateiz mateiz added a commit that referenced this pull request Mar 11, 2013

@mateiz mateiz Merge pull request #513 from MLnick/bagel-caching
Adds choice of persistence level to Bagel.
cbf8f0d

@mateiz mateiz merged commit cbf8f0d into mesos:master Mar 11, 2013

Owner

mateiz commented Mar 11, 2013

Cool, thanks!

MLnick deleted the unknown repository branch Mar 11, 2013

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment