New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SPARK-29670][CORE][2.4] Make executor bind address configurable for Spark 2.4 #32633
Conversation
Can one of the admins verify this patch? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This could be OK, but I'm not sure there will be any more 2.4.x releases after 2.4.8 anyway
24b4511
to
99c360c
Compare
Ow, I didn't aware of this. That's explains |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you for making a PR, @gedeh .
However, this PR is invalid in a few ways.
- SPARK-29670 is closed as
Duplicated
. You cannot ask a backport of unmerged JIRA. - The actual merged JIRA for feature is SPARK-24203 which was an
Improvement
at Apache Spark 3.0.0. Apache Spark community have a general policy which allows bug-fix backporting only. So, we cannot backport a new feature because we need to protect old branches. - Lastly, 2.4.8 was the EOL release already. Here is some explanation for that.
The last minor release within a major a release will typically be maintained for longer as an “LTS” release. For example, 2.4.0 was released in November 2nd 2018 and had been maintained for 31 months until 2.4.8 was released on May 2021. 2.4.8 is the last release and no more 2.4.x releases should be expected even for bug fixes.
Given that the above reasons, could you close this PR, @gedeh ?
Thank you @dongjoon-hyun for clarification. I wasn't aware of EOL of Spark 2.4 closing this PR. Thanks! |
Thank you so much, @gedeh ! |
What changes were proposed in this pull request?
Backporting changes from #26331 to Spark 2.4, in a similar fashion, executor's
bindAddress
is passed as an input parameter forRPCEnv.create
.This PR also ensure
YarnAllocator
use the config when allocating new executor and use default hostname when its not configured.Why are the changes needed?
Recently we've came across this issue with Spark running on Yarn in Istio enabled Kubernetes cluster. As I understand it, Spark 2.4 is a stepping stone for teams moving away from Scala 2.11 to Scala 2.12 and then moving to Spark 3.
Does this PR introduce any user-facing change?
Yes, new config is added and relevant doc is updated.
How was this patch tested?
I will run this on Kubenetes and Istio.