[SPARK-33029][CORE][WEBUI] Fix the UI executor page incorrectly marking the driver as excluded #30954
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
cc @tgravescs |
|
Test build #133458 has finished for PR 30954 at commit
|
|
To retriever GitHub Action, could you rebase to the master branch, @baohe-zhang ? |
Done. |
|
it looks like you need to fix up the test: [info] - executor node excludeOnFailure *** FAILED *** (220 milliseconds) I think we had the file have the driver as excluded |
|
@tgravescs The unit tests are fixed. |
|
Kubernetes integration test starting |
|
Test build #133623 has finished for PR 30954 at commit
|
|
Kubernetes integration test status success |
tgravescs
approved these changes
Jan 5, 2021
|
Thank you, @baohe-zhang and @tgravescs . |
|
Thanks @dongjoon-hyun. cc @gengliangwang and @sarutak FYI |
dongjoon-hyun
approved these changes
Jan 6, 2021
dongjoon-hyun
pushed a commit
that referenced
this pull request
Jan 6, 2021
…ng the driver as excluded ### What changes were proposed in this pull request? Filter out the driver entity when updating the exclusion status of live executors(including the driver), so the UI won't be marked as excluded in the UI even if the node that hosts the driver has been marked as excluded. ### Why are the changes needed? Before this change, if we run spark with the standalone mode and with spark.blacklist.enabled=true. The driver will be marked as excluded when the host that hosts that driver has been marked as excluded. While it's incorrect because the exclude list feature will exclude executors only and the driver is still active.  After the fix, the driver won't be marked as excluded.  ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Manual test. Reopen the UI and see the driver is no longer marked as excluded. Closes #30954 from baohe-zhang/SPARK-33029. Authored-by: Baohe Zhang <baohe.zhang@verizonmedia.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com> (cherry picked from commit 2951082) Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
|
Could you make a backport to branch-3.0, @baohe-zhang ? |
|
@dongjoon-hyun Sure I will make a backport. |
|
Thanks! |
dongjoon-hyun
pushed a commit
that referenced
this pull request
Jan 6, 2021
…marking the driver as blacklisted This is a backport of #30954 ### What changes were proposed in this pull request? Filter out the driver entity when updating the exclusion status of live executors(including the driver), so the driver won't be marked as blacklisted in the UI even if the node that hosts the driver has been marked as blacklisted. ### Why are the changes needed? Before this change, if we run spark with the standalone mode and with spark.blacklist.enabled=true. The driver will be marked as blacklisted when the host that hosts that driver has been marked as blacklisted. While it's incorrect because the exclude list feature will exclude executors only and the driver is still active.  After the fix, the driver won't be marked as blacklisted.  ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Manual test. Reopen the UI and see the driver is no longer marked as blacklisted. Closes #31057 from baohe-zhang/SPARK-33029-3.0. Authored-by: Baohe Zhang <baohe.zhang@verizonmedia.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Filter out the driver entity when updating the exclusion status of live executors(including the driver), so the UI won't be marked as excluded in the UI even if the node that hosts the driver has been marked as excluded.
Why are the changes needed?
Before this change, if we run spark with the standalone mode and with spark.blacklist.enabled=true. The driver will be marked as excluded when the host that hosts that driver has been marked as excluded. While it's incorrect because the exclude list feature will exclude executors only and the driver is still active.
After the fix, the driver won't be marked as excluded.
Does this PR introduce any user-facing change?
No.
How was this patch tested?
Manual test. Reopen the UI and see the driver is no longer marked as excluded.