-
Notifications
You must be signed in to change notification settings - Fork 74
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Improvement] Only initialize the heartbeat thread pool when the role is driver #177
Conversation
The spark2 also have similar problems. Could you modify the spark2 at the same time? |
Updated @jerqi . I'm confused that when heartbeat failed, it will do nothing. Right? |
Heartbeat is to notify shuffle server app is alive, and every rpc also will do the notify. According to current implementation, ignore the failed rpc. |
If shuffle server hasn't received any heartbeat of application, shuffle server will delete the shuffle data of the application. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM, Thanks for your contribution.
backport 0.5.0 ### What changes were proposed in this pull request? We need to judge heartbeatExecutorService whether is null when we will stop it. ### Why are the changes needed? #177 pr introduce this problem, when we run Spark applications on our cluster, the executor will throw NPE when method `stop` is called. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Manual test
### What changes were proposed in this pull request? We need to judge heartbeatExecutorService whether is null when we will stop it. ### Why are the changes needed? #177 pr introduce this problem, when we run Spark applications on our cluster, the executor will throw NPE when method `stop` is called. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Manual test
backport 0.5.0 ### What changes were proposed in this pull request? We need to judge heartbeatExecutorService whether is null when we will stop it. ### Why are the changes needed? #177 pr introduce this problem, when we run Spark applications on our cluster, the executor will throw NPE when method `stop` is called. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Manual test
What changes were proposed in this pull request?
Only initialize the heartbeat thread pool when the role is driver
Why are the changes needed?
Just to avoid creating extra thread pool.
Does this PR introduce any user-facing change?
No
How was this patch tested?
No