[SPARK-55134] Fix BasicExecutorFeatureStep to throw IllegalArgumentException for executor cpu misconfigs

[dongjoon-hyun]

What changes were proposed in this pull request?

This PR aims to fix BasicExecutorFeatureStep to throw IllegalArgumentException for executor cpu misconfigs in order the Spark jobs ASAP.

Why are the changes needed?

From Apache Spark 4.1.0, Spark driver pod throws SparkException for the executor cpu misconfiguration before requesting to the K8s control plain. This improvement reduces the burden of K8s control plane.

• [SPARK-52933][K8S] Verify if the executor cpu request exceeds limit #51678

26/01/24 06:55:31 INFO ExecutorPodsAllocator: Going to request 5 executors from Kubernetes for ResourceProfile Id: 0, target: 5, known: 0, sharedSlotFromPendingPods: 2147483647.
26/01/24 06:55:31 INFO ExecutorPodsAllocator: Found 0 reusable PVCs from 0 PVCs
26/01/24 06:55:31 WARN ExecutorPodsSnapshotsStoreImpl: Exception when notifying snapshot subscriber.
org.apache.spark.SparkException: The executor cpu request (4) should be less than or equal to cpu limit (1)
	at org.apache.spark.deploy.k8s.features.BasicExecutorFeatureStep.$anonfun$configurePod$11(BasicExecutorFeatureStep.scala:236)

However, the Spark driver keeps re-trying to create executor pods in any way if the users didn't have an additional spark.driver.timeout configuration.

• [SPARK-47207][CORE] Support spark.driver.timeout and DriverTimeoutPlugin #45313

So, we had better exit the Spark job in this case ASAP. We can do that simply switches SparkException to IllegalArgumentException like the other steps.

• [SPARK-33175][K8S] Detect duplicated mountPaths and fail at Spark side #30084

Does this PR introduce any user-facing change?

Technically no because previously those misconfigured Spark job didn't get any resources.

How was this patch tested?

Pass the CIs with the updated test case.

Also, I checked manually via spark-submit:

$ bin/spark-submit --master k8s://$K8S_MASTER \
--deploy-mode cluster \
-c spark.executor.instances=5 \
-c spark.kubernetes.executor.request.cores=4 \
-c spark.kubernetes.executor.limit.cores=1 \
-c spark.kubernetes.container.image=apache/spark:SPARK-55134 \
-c spark.kubernetes.authenticate.driver.serviceAccountName=spark \
-c spark.kubernetes.executor.useDriverPodIP=true \
--class org.apache.spark.examples.SparkPi \
local:///opt/spark/examples/jars/spark-examples.jar 200000
...
26/01/24 16:33:57 INFO LoggingPodStatusWatcherImpl: State changed, new state:
	 pod name: org-apache-spark-examples-sparkpi-0482f19beeec7491-driver
	 namespace: default
	 labels: spark-app-name -> org-apache-spark-examples-sparkpi, spark-app-selector -> spark-ee23f03db88b43fb906b0dbc1b04ad63, spark-role -> driver, spark-version -> 4.2.0-SNAPSHOT
	 pod uid: c6d41845-5893-4135-a065-278d94500315
	 creation time: 2026-01-24T07:33:52Z
	 service account name: spark
	 volumes: spark-local-dir-1, spark-conf-volume-driver, kube-api-access-8rbc8
	 node name: lima-rancher-desktop
	 start time: 2026-01-24T07:33:52Z
	 phase: Failed
	 container status:
		 container name: spark-kubernetes-driver
		 container image: apache/spark:SPARK-55134
		 container state: terminated
		 container started at: 2026-01-24T07:33:53Z
		 container finished at: 2026-01-24T07:33:55Z
		 exit code: 1
		 termination reason: Error
26/01/24 16:33:57 INFO LoggingPodStatusWatcherImpl: Application status for spark-ee23f03db88b43fb906b0dbc1b04ad63 (phase: Failed)
26/01/24 16:33:57 INFO LoggingPodStatusWatcherImpl: Container final statuses:
	 container name: spark-kubernetes-driver
	 container image: apache/spark:SPARK-55134
	 container state: terminated
	 container started at: 2026-01-24T07:33:53Z
	 container finished at: 2026-01-24T07:33:55Z
	 exit code: 1
	 termination reason: Error
26/01/24 16:33:57 INFO LoggingPodStatusWatcherImpl: Application org.apache.spark.examples.SparkPi with application ID spark-ee23f03db88b43fb906b0dbc1b04ad63 and submission ID default:org-apache-spark-examples-sparkpi-0482f19beeec7491-driver finished

Was this patch authored or co-authored using generative AI tooling?

No.

[github-actions]

JIRA Issue Information

=== Bug SPARK-55134 ===
Summary: If the executor’s resource request is greater than its limit, the driver is expected to exit.
Assignee: None
Status: Open
Affected: ["4.1.0","4.1.1"]

---

This comment was automatically generated by GitHub Actions

[dongjoon-hyun]

Could you review this PR when you have some time, @peter-toth ?

[peter-toth]

Sure. The change looks good to me.

[dongjoon-hyun]

Thank you so much, @peter-toth !

Merged to master/4.1.