New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bagel: Large-scale graph processing on Spark #48

Merged
merged 10 commits into from May 13, 2011

Conversation

Projects
None yet
3 participants
@ankurdave
Member

ankurdave commented May 3, 2011

Bagel is an implementation of the Pregel graph processing framework on Spark.

Bagel currently supports basic graph computation, combiners, and aggregators. Future work includes support for mutating the graph topology. Tests exist but currently don't run due to a Spark bug.

ankurdave added some commits Apr 13, 2011

Add Bagel test suite
Note: This test suite currently fails for the same reason that the
Spark Core test suite fails: Spark currently seems to have a bug where
any test after the first one fails.
Refactor and add aggregator support
Refactored out the agg() and comp() methods from Pregel.run.

Defined an implicit conversion to allow applications that don't use
aggregators to avoid including a null argument for the result of the
aggregator in the compute function.
@tjhunter

This comment has been minimized.

Show comment
Hide comment
@tjhunter

tjhunter May 4, 2011

I would recommend you refactor your code before merging, it is always harder / less tempting to do after.

tjhunter commented May 4, 2011

I would recommend you refactor your code before merging, it is always harder / less tempting to do after.

@mateiz

This comment has been minimized.

Show comment
Hide comment
@mateiz

mateiz May 9, 2011

Member

This looks great, Ankur, except for two naming things: can you change the package name from bagel to spark.bagel, and can you rename the Pregel class to Bagel?

Member

mateiz commented May 9, 2011

This looks great, Ankur, except for two naming things: can you change the package name from bagel to spark.bagel, and can you rename the Pregel class to Bagel?

@ankurdave

This comment has been minimized.

Show comment
Hide comment
@ankurdave

ankurdave May 9, 2011

Member

Sure, I've done so.

Member

ankurdave commented May 9, 2011

Sure, I've done so.

mateiz added a commit that referenced this pull request May 13, 2011

Merge pull request #48 from ankurdave/bagel-new
Bagel: Large-scale graph processing on Spark

@mateiz mateiz merged commit 4b1f0f1 into mesos:new-rdds May 13, 2011

@mateiz

This comment has been minimized.

Show comment
Hide comment
@mateiz

mateiz May 13, 2011

Member

Looks great, thanks. The one thing I should add is that maybe you should write a README documenting the examples, or a wiki page (and put a comment in the code pointing to this location).

Member

mateiz commented May 13, 2011

Looks great, thanks. The one thing I should add is that maybe you should write a README documenting the examples, or a wiki page (and put a comment in the code pointing to this location).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment