[SPARK-45278] [YARN] Allow configuring Yarn executor bind address in Yarn

[gedeh]

What changes were proposed in this pull request?

Uptake --bind-address parameter in YarnCoarseGrainedExecutorBackend when launching new container in Yarn cluster. This PR also ensure YarnAllocator uses default hostname when its not configured.

Why are the changes needed?

We've came across istio/istio#27900 with Spark running on Yarn in Istio enabled Kubernetes cluster. Previous PR #32633 is not merged because Spark 2.4 was EOL.

Does this PR introduce any user-facing change?

Yes, new config specifically for Yarn cluster mode is added and relevant doc is updated.

How was this patch tested?

Tested in Kubenetes with Istio and added tests to YarnAllocatorSuite

Thanks!

[gedeh]

Apologies using old Jira because I am unable to raise new Spark Jira in ASF

[gedeh]

Latest full test run via Github Action https://github.com/gedeh/spark/actions/runs/6143764745

[gedeh]

@srowen and @dongjoon-hyun you were on previous PR #32633
Apologies I can't raise new Jira in Spark project for this issue

[srowen]

Can you remove the unrelated changes?
Do you have a JIRA account? You can request one on the JIRA site.

[gedeh]

Thank you for the prompt reply @srowen

Can you remove the unrelated changes?

Which changes these will be? Are you suggesting the changes in docs/configuration.md?

Do you have a JIRA account? You can request one on the JIRA site.

I requested an account few days ago, but unfortunately the request rejected

[gedeh]

@srowen got my ASF account created and raised this Jira https://issues.apache.org/jira/browse/SPARK-45278. Removed the change in configuration.md file too. Thanks in advance

[tokoko]

Why YARN cluster mode? The change affects client mode as well, doesn't it?

[gedeh]

Hmm true, will benefit client mode as well

[github-actions]

We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!

[gedeh]

Hello, I noticed this PR is closed, this is blocking Spark with Yarn in kubernetes. I dont understand whats left missing for this PR. If anyone in Spark project can shed a light what required to merge this for next release of Spark that would be great. Thanks!

[laocoder2014]

I think this PR is needed for Spark to work under Istio/service mesh. I am just wondering if anyone has run Spark/YARN successfully under Istio/Service Mesh.
The PR Spark-26331 was merged and made the 'executor's bindAddress' configurable. But without the above PR, how can user configure the executor's bindAddress?
@srowen @tokoko Can you look into this PR? Can you reopen/merge it?

[gedeh]

I think this PR is needed for Spark to work under Istio/service mesh. I am just wondering if anyone has run Spark/YARN successfully under Istio/Service Mesh. The PR Spark-26331 was merged and made the 'executor's bindAddress' configurable. But without the above PR, how can user configure the executor's bindAddress? @srowen @tokoko Can you look into this PR? Can you reopen/merge it?

I've created new PR #47892