[SPARK-58113][K8S] Support publishNotReadyAddresses in driver service

[j1wonpark]

What changes were proposed in this pull request?

Add a new configuration spark.kubernetes.driver.service.publishNotReadyAddresses (boolean, default false).

When enabled:

• DriverServiceFeatureStep sets spec.publishNotReadyAddresses: true on the headless driver service, so DNS records for the driver pod are published even while the pod is not Ready.
• The driver pod readiness wait before executor allocation (introduced in SPARK-32975) is skipped in ExecutorPodsAllocator, StatefulSetPodsAllocator, and DeploymentPodsAllocator, since the sole purpose of that wait — ensuring the headless service is resolvable by DNS before executors start — no longer depends on pod readiness.

Why are the changes needed?

Kubernetes publishes DNS records for a headless service only for Ready pods. Users commonly attach a readiness probe to the driver pod through spark.kubernetes.driver.podTemplateFile — a typical setup for long-running Spark Connect servers, where pod readiness gates traffic routing. When such a probe gates on a port that binds after SparkContext initialization completes (e.g. the Spark Connect gRPC port), executors launched during initialization cannot resolve the driver service and die with UnknownHostException. With static allocation, maxNumExecutorFailures = max(3, 2 * spark.executor.instances) is reached before initialization can complete, and the driver exits with code 11 (EXCEED_MAX_EXECUTOR_FAILURES) on every attempt — the application can never start.

The existing mitigation from SPARK-32975 (waitUntilReady in the allocators, tunable via spark.kubernetes.allocation.driver.readinessTimeout per SPARK-49079) cannot help here: the wait runs inside SparkContext initialization, while such a probe can only pass after initialization completes, so the wait always burns the full timeout and the race resumes.

The driver service is a single-pod discovery endpoint rather than a load-balancing service, and publishNotReadyAddresses is the standard Kubernetes pattern for headless discovery services. There is currently no first-class way to set it — the service spec is not customizable via pod templates, and the only workarounds are re-implementing the service in a custom feature step (via spark.kubernetes.driver.pod.excludedFeatureSteps) or cluster-level admission webhooks. This follows the precedent of exposing driver service spec fields as configurations (SPARK-39490: ipFamilyPolicy/ipFamilies).

The configuration is opt-in (default false) because publishing not-ready addresses is not desirable for deployments that rely on endpoint readiness to gate traffic to the driver service (e.g. through a service mesh); flipping the default could be discussed separately.

Does this PR introduce any user-facing change?

No change by default. A new opt-in configuration is added and documented in running-on-kubernetes.md.

How was this patch tested?

New unit tests:

• DriverServiceFeatureStepSuite: the default leaves the field false; enabling the configuration sets it to true.
• ExecutorPodsAllocatorSuite, StatefulSetAllocatorSuite, DeploymentAllocatorSuite: the driver pod readiness wait still runs by default, and enabling the configuration skips it (waitUntilReady is not invoked).

Also verified the behavior end-to-end on a real Kubernetes cluster by applying publishNotReadyAddresses: true to the driver service through an admission webhook (equivalent to what this configuration does): executors resolved the driver service and registered while the driver pod was still NotReady behind a Spark Connect port readiness probe, and a static-allocation application that previously failed on every attempt with exit code 11 started successfully.

Was this patch authored or co-authored using generative AI tooling?

Yes. Co-authored with Claude Opus.

[dongjoon-hyun]

Thank you for making a PR, @j1wonpark . The code looks reasonable to me. Please add a binding policy.

[uros-b]

Note: the rationale "so executors can resolve the driver service during startup" only holds when spark.kubernetes.executor.useDriverPodIP=false. That config defaults to true (since 4.1.0+), where the executor's driverAddress is DRIVER_BIND_ADDRESS (the pod IP) and executors never DNS-resolve the driver service; so in the default configuration this option, and the readiness wait it bypasses, are moot. A user reading only this entry may enable it believing it helps the default setup. One clarifying sentence (only meaningful when useDriverPodIP=false) in both the config doc and the docs table would close the gap.

[uros-b]

Overall looks good, thank you @j1wonpark - @dongjoon-hyun PTAL, also cc @cchriswu @pan3793

[dongjoon-hyun]

+1, LGTM.

[dongjoon-hyun]

Merge Summary:

• merged into master 01b3c25
• merged into branch-4.x fad3305

Posted by merge_spark_pr.py