[SPARK-55653][K8S] Support NetworkPolicy for Spark executor pods

[dongjoon-hyun]

What changes were proposed in this pull request?

This PR aims to support NetworkPolicy for Spark executor pods.

Why are the changes needed?

NetworkPolicy is frequently used in the production to isolate Spark applications.

• https://kubernetes.io/docs/concepts/services-networking/network-policies/

Does this PR introduce any user-facing change?

This is a security feature to make Spark K8s executor pods access only from the pods with the same application ID.

There are two ways if a user wants to access the executor pods from outside.

1. Use a pod with the same application ID with the target Spark applications.
2. Submit a Spark job with the following configuration.

spark.kubernetes.driver.pod.excludedFeatureSteps=org.apache.spark.deploy.k8s.features.NetworkPolicyFeatureStep

How was this patch tested?

Pass the CIs with the newly added test suite.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Gemini 3.1 Pro (High) on Antigravity

[dongjoon-hyun]

cc @peter-toth

[peter-toth]

Does this change need to be explicitly mentioned in migration docs?

[dongjoon-hyun]

Thank you, @peter-toth . I addressed your comment by updating the migration guide.

• Since Spark 4.2, Spark configures a NetworkPolicy by default so that executor pods only accept ingress traffic from the driver and peer executors within the same job. To disable this and restore the legacy behavior, set spark.kubernetes.driver.pod.excludedFeatureSteps to org.apache.spark.deploy.k8s.features.NetworkPolicyFeatureStep.

[dongjoon-hyun]

Merged to master