Bagel: Large-scale graph processing on Spark #48

Merged
merged 10 commits into from May 13, 2011

Projects

None yet

3 participants

@ankurdave
Member

Bagel is an implementation of the Pregel graph processing framework on Spark.

Bagel currently supports basic graph computation, combiners, and aggregators. Future work includes support for mutating the graph topology. Tests exist but currently don't run due to a Spark bug.

ankurdave added some commits Apr 13, 2011
@ankurdave ankurdave Add Bagel, an implementation of Pregel on Spark c0736f6
@ankurdave ankurdave Clean up Pregel.run, add logging 62ef620
@ankurdave ankurdave Add Bagel classpath to run script 45ec9db
@ankurdave ankurdave Update ShortestPath to work with controllable partitioning 19122af
@ankurdave ankurdave Clean up Bagel source and interface c5b3ea7
@ankurdave ankurdave Add Bagel test suite
Note: This test suite currently fails for the same reason that the
Spark Core test suite fails: Spark currently seems to have a bug where
any test after the first one fails.
1c8ca0e
@ankurdave ankurdave Package combiner functions into a trait c18fa3e
@ankurdave ankurdave Refactor and add aggregator support
Refactored out the agg() and comp() methods from Pregel.run.

Defined an implicit conversion to allow applications that don't use
aggregators to avoid including a null argument for the result of the
aggregator in the compute function.
563c5e7
@tjhunter
tjhunter commented May 4, 2011

I would recommend you refactor your code before merging, it is always harder / less tempting to do after.

@mateiz
Member
mateiz commented May 9, 2011

This looks great, Ankur, except for two naming things: can you change the package name from bagel to spark.bagel, and can you rename the Pregel class to Bagel?

@ankurdave
Member

Sure, I've done so.

@mateiz mateiz merged commit 4b1f0f1 into mesos:new-rdds May 13, 2011
@mateiz
Member
mateiz commented May 13, 2011

Looks great, thanks. The one thing I should add is that maybe you should write a README documenting the examples, or a wiki page (and put a comment in the code pointing to this location).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment