Bagel is an implementation of the Pregel graph processing framework on Spark.
Bagel currently supports basic graph computation, combiners, and aggregators. Future work includes support for mutating the graph topology. Tests exist but currently don't run due to a Spark bug.
Add Bagel, an implementation of Pregel on Spark
Clean up Pregel.run, add logging
Add Bagel classpath to run script
Update ShortestPath to work with controllable partitioning
Clean up Bagel source and interface
Add Bagel test suite
Note: This test suite currently fails for the same reason that the
Spark Core test suite fails: Spark currently seems to have a bug where
any test after the first one fails.
Package combiner functions into a trait
Refactor and add aggregator support
Refactored out the agg() and comp() methods from Pregel.run.
Defined an implicit conversion to allow applications that don't use
aggregators to avoid including a null argument for the result of the
aggregator in the compute function.
I would recommend you refactor your code before merging, it is always harder / less tempting to do after.
Move shortest path and PageRank to bagel.examples
This looks great, Ankur, except for two naming things: can you change the package name from bagel to spark.bagel, and can you rename the Pregel class to Bagel?
Rename bagel to spark.bagel and Pregel to Bagel
Sure, I've done so.
Looks great, thanks. The one thing I should add is that maybe you should write a README documenting the examples, or a wiki page (and put a comment in the code pointing to this location).