forked from apache/spark
-
Notifications
You must be signed in to change notification settings - Fork 51
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SPARK-25299] ShuffleLocation/FetchFailed integrations with scheduler #548
Closed
Closed
Changes from all commits
Commits
Show all changes
27 commits
Select commit
Hold shift + click to select a range
36d4898
semantic fix that hasn't merged yet (to revert)
yifeih 48f52c8
specify ShuffleLocation interface
yifeih d161feb
more api changes
yifeih 070151b
wip
yifeih 190e2de
fix
yifeih a34a103
style
yifeih 93c2449
fix style, fix test
yifeih fcb6ac7
cleanup
yifeih 715a7ed
use seq in map output tracker
yifeih b68254c
always remove for remote
yifeih df8d457
fix some tests
yifeih 97f6ff3
json serialize
yifeih 09eeb13
array -> seq
yifeih ba78f9a
fix serialization
yifeih f50a309
fix javadoc
yifeih b7c1e2a
write async scheduler tests
yifeih de824d4
fix the async location tests
yifeih 7794b68
add tests for distributed file server case
yifeih 9604889
use scala in tests
yifeih 58c8ef2
more buffers
yifeih 506a3f7
use Seq.empty instead of null
yifeih 118a2bf
add flag for removing other map statuses
yifeih bf8aeab
small fixes
yifeih 994ff5a
use seq empty
yifeih 7758a90
simplify async case
yifeih cebe386
remove another null to Seq.empty
yifeih 056c87b
one more
yifeih File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I mentioned this on the doc, but I'm skeptical about supporting different locations for each (map, reduce) block, instead of just replicating the entire output of one map task to the same places. I don't think I properly understood that part even before this change ... I'll need to look through this more carefully to figure out what the effect of that would be, in particular how much bookkeeping is required on the driver.