[SPARK-51347] Enable Ingress and Service Support for Spark Driver

[jiangzho]

What changes were proposed in this pull request?

This PR adds support for launching Ingress and Services with Spark Applications.

Why are the changes needed?

There's common ask from user to support exposing driver via ingress and service to serve HTTP / HTTPS from outside the cluster - for example, to access Spark UI.

As we support podTemplateSupport already, this feature can also be handy for those who would like to expose additional service ports in the same pod. Therefore, this PR aims to introduce this feature in a more general way instead of just exposing Spark UI.

Does this PR introduce any user-facing change?

No - unreleased version

How was this patch tested?

CIs

Was this patch authored or co-authored using generative AI tooling?

No

[dongjoon-hyun]

Thank you for making a PR, @jiangzho .

[dongjoon-hyun]

+1, LGTM. Merged to main.

[dongjoon-hyun]

BTW, I noticed that this is a first commit with a different email, @jiangzho . Are you going to use this one?

$ git log --author=zh | grep 'Author:' | sort | uniq -c
   1 Author: Zhou JIANG <jiangzho@umich.edu>
  20 Author: zhou-jiang <zhou_jiang@apple.com>

[jiangzho]

Thanks for the reminder! This is a result of local squashing, I'll fix this for following commits - will make it consist to use jiangzho@umich.edu