Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pipelined Execution #16

Open
anilpacaci opened this issue Jan 16, 2020 · 2 comments
Open

Pipelined Execution #16

anilpacaci opened this issue Jan 16, 2020 · 2 comments

Comments

@anilpacaci
Copy link
Owner

This issue is related to #15 . We can maintain a buffer for incoming edges in each partition, then we can potentially parallelize execution of multiple stream tuples. Here each buffer needs to maintain FIFO order to ensure correctness within a spanning tree.

@anilpacaci
Copy link
Owner Author

This is indeed an easy way to parallelize the execution of multiple edges. Spanning trees do not need to communicate and as long as the order preserved within a spanning tree, we can provide correct answers. The challenge is load balancing as some spanning trees will get extremely large and do most of the work

@anilpacaci
Copy link
Owner Author

A real interesting/challenging parallel execution is the scenario where multiple threads work on the same tree. Here we need to talk about serializability, whenever a thread Xinserts an edge E to the tree, it needs to make sure that other threads that started before X for a transition before this edge E has already completed. This ensures that edge E has all the source nodes placed in the tree when X starts processing

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant