[SPARK-49838] Add spark-version label to Spark Cluster resources

[dongjoon-hyun]

What changes were proposed in this pull request?

This PR aims to add spark-version label to Spark Cluster resources like Spark Application

Why are the changes needed?

spark-version is an important label to distinguish and search in the production environment. This PR will add spark-version to the following resources.

• Master Service
• Master Statefulset
• Master Pod
• Worker Service
• Worker Statefulset
• Worker HorizontalPodAutoscaler
• Worker Pod

We can use spark-version as a selector like the following.

$ kubectl get svc
NAME                          TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)                      AGE
cluster-with-hpa-master-svc   ClusterIP   None         <none>        8080/TCP,7077/TCP,6066/TCP   39s
cluster-with-hpa-worker-svc   ClusterIP   None         <none>        8081/TCP                     39s
kubernetes                    ClusterIP   10.96.0.1    <none>        443/TCP                      6d12h

$ kubectl get svc -l spark-version=4.0.0-preview2
NAME                          TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)                      AGE
cluster-with-hpa-master-svc   ClusterIP   None         <none>        8080/TCP,7077/TCP,6066/TCP   89s
cluster-with-hpa-worker-svc   ClusterIP   None         <none>        8081/TCP                     89s

$ kubectl get pod -l spark-version=4.0.0-preview2
NAME                        READY   STATUS    RESTARTS   AGE
cluster-with-hpa-master-0   1/1     Running   0          17m
cluster-with-hpa-worker-0   1/1     Running   0          17m

Does this PR introduce any user-facing change?

No, this is not released yet.

How was this patch tested?

Pass the CIs with the revised test cases.

Was this patch authored or co-authored using generative AI tooling?

No.

[dongjoon-hyun]

cc @viirya

[dongjoon-hyun]

Thank you, @viirya . Merged to main.