-
Notifications
You must be signed in to change notification settings - Fork 28.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SPARK-19753][CORE] Un-register all shuffle output on a host in case of slave lost or fetch failure #18150
Closed
Closed
[SPARK-19753][CORE] Un-register all shuffle output on a host in case of slave lost or fetch failure #18150
Changes from all commits
Commits
Show all changes
16 commits
Select commit
Hold shift + click to select a range
4643325
[SPARK-19753][CORE] All shuffle files on a host should be removed in …
6145d2a
Do not un-register shuffle files in case of executor lost
ef86d12
no-op when external shuffle service is disabled
263d55b
fix check style
a548dc3
Addressed review comments and fixed a bug
b5f1949
Fix build
705a5aa
Fix test failures
996ad20
refactoring & comments
squito 78f9094
Review comments
3b53f7e
Minor changes as per review comments
e5f532e
Make the behaviour configurable and minor review comments
1189e40
Minor changes
9d700e2
set the behavior to false by defaut
b26e99d
Fix test
74f285a
Rebase with master
ba2ca2a
remove extra while space
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
one more question: if worker lost, shouldn't we unregister outputs on that worker/host?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
seems we can't get worker id here, nvm