[SPARK-36061][K8S] Add volcano built-in module and feature step

[Yikun]

What changes were proposed in this pull request?

This patch added volcano feature step to help user integrate spark with Volcano Scheduler.

• Add a VolcanoFeatureStep, it can be used in driver and executor side.

After this patch, users can enable this featurestep by submiting job by using

--conf spark.kubernete.driver.scheduler.name=volcano \
--conf spark.kubernetes.driver.pod.featureSteps=org.apache.spark.deploy.k8s.features.scheduler.VolcanoFeatureStep

A PodGroup will be created before driver started, annotations will be set to driver pod to added driver pod to this pod group. Then, Volcano scheduler will help driver pod scheduling instead of deafult kubernetes scheduler.

Why are the changes needed?

This PR help user integrate Spark with Volcano Scheduler.

See also: SPARK-36057

Does this PR introduce any user-facing change?

Yes, introduced a user feature step.

These are used by VolcanoFeatureStep, and also will be used by YunikornFeatureStep in future.

How was this patch tested?

• UT
• Integration test: Test all VolcanoSuite (all kubernetes test with volcano + a new podgroup test) and KubernetesSuite

# Deploy Volcano (x86)
kubectl apply -f https://raw.githubusercontent.com/volcano-sh/volcano/master/installer/volcano-development.yaml
# Deploy Volcano (arm64)
kubectl apply -f https://raw.githubusercontent.com/volcano-sh/volcano/master/installer/volcano-development-arm64.yaml
# Test all VolcanoSuite (all kubernetes test with volcano + a new podgroup test) and KubernetesSuite
build/sbt -Pkubernetes -Pkubernetes-integration-tests -Dtest.exclude.tags=minikube,r "kubernetes-integration-tests/test"

[HyukjinKwon]

Yeah .. I prefer this way but would be good to have some feedback from people like @dongjoon-hyun as you mentioned in another PR.

[dongjoon-hyun]

Sorry but -1 for this embedding this experimental dependency into K8s GA module and the default distribution.

If we add this once, we cannot remove this, @HyukjinKwon and @Yikun .

[dongjoon-hyun]

This feature should be marked as Experimental in the code and documents at Apache Spark 3.3.

[dongjoon-hyun]

One more thing. In the original PR, @Yikun proposed as a tightly coupled style initially and we revised it into the recent form to be better (more neutral and extensible) in the architectural form.

[HyukjinKwon]

I am fine. I would defer to @dongjoon-hyun here since he has more background on K8S.

[dongjoon-hyun]

Thank you, @HyukjinKwon . I'm trying to deliver this feature safely in a step-wise manner. To do that, I'm also actively testing all relevant features in the production environments.

[Yikun]

Thanks for your discussion and suggestion! I'm also defer to @dongjoon-hyun and @HyukjinKwon decision, so I will close this PR, and continue on profile module PR.

I'm trying to deliver this feature safely in a step-wise manner. To do that, I'm also actively testing all relevant features in the production environments.

@dongjoon-hyun Thanks for taking care this!