[SYSTEMDS-3466] Asynchronous (Future-based) execution of Spark instructions #1733
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…ctions This patch introduces a future-based asynchronous execution of a Spark actions. We wrap the matrix block with a future, create a matrix object handle, and maintain that in the symbol table. This extension allows triggering a chain of Spark instructions asynchronously and seek the results only when needed. TODO: Account the memory required for the future results, maintain lineage of the broadcast variables to avoid premature removal.
This is really interesting @phaniarnab . Curious: is there a concrete use case for this feature? |
Yes. There are different use cases. They arise as we combine Spark's lazy execution with local eager execution. Currently, a chain of Spark operations is triggered only when we need the result locally---which leads to inefficient cluster utilization. @BACtaki |
fathollahzadeh
pushed a commit
to fathollahzadeh/systemds
that referenced
this pull request
Dec 7, 2022
…ctions This patch introduces a future-based asynchronous execution of Spark actions. We wrap the matrix block with a future, create a matrix object handle, and maintain that in the symbol table. This extension allows triggering a chain of Spark instructions asynchronously and seeking the results only when needed. TODO: Account the memory required for the future results, maintain lineage of the broadcast variables to avoid premature removal. Closes apache#1733
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This patch introduces a future-based asynchronous execution of Spark actions. We wrap the matrix block with a future, create a matrix object handle, and maintain that in the symbol table. This extension allows triggering a chain of Spark instructions asynchronously and seeking the results only when needed.
TODO: Account the memory required for the future results, maintain the lineage of the broadcast variables to avoid premature removal.