Skip to content

[SPARK-55757][CORE] Improve spark.task.cpus validation#54559

Closed
dongjoon-hyun wants to merge 1 commit intoapache:masterfrom
dongjoon-hyun:SPARK-55757
Closed

[SPARK-55757][CORE] Improve spark.task.cpus validation#54559
dongjoon-hyun wants to merge 1 commit intoapache:masterfrom
dongjoon-hyun:SPARK-55757

Conversation

@dongjoon-hyun
Copy link
Member

@dongjoon-hyun dongjoon-hyun commented Feb 28, 2026

What changes were proposed in this pull request?

This PR aim to improve spark.task.cpus validation.

Why are the changes needed?

Currently, Apache Spark throws java.lang.ArithmeticException for the invalid spark.task.cpus. We had better provide actionable direction.

BEFORE

$ bin/spark-shell -c spark.task.cpus=0
26/02/28 09:00:56 ERROR SparkContext: Error initializing SparkContext.
java.lang.ArithmeticException: / by zero
	at org.apache.spark.resource.ResourceUtils$.warnOnWastedResources(ResourceUtils.scala:444)

AFTER

$ bin/spark-shell -c spark.task.cpus=0
...
26/02/28 09:02:15 ERROR SparkContext: Error initializing SparkContext.
org.apache.spark.SparkIllegalArgumentException: [INVALID_CONF_VALUE.REQUIREMENT] The value '0' in the config "spark.task.cpus" is invalid. The number of task CPUs must be positive. SQLSTATE: 22022

Does this PR introduce any user-facing change?

This only changes the error message before starting Spark job.

How was this patch tested?

Pass the CIs with newly added test case.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Gemini 3.1 Pro (High) on Antigravity.

@dongjoon-hyun
Copy link
Member Author

Thank you, @pan3793 .

@dongjoon-hyun
Copy link
Member Author

Also, FYI, cc @peter-toth

@dongjoon-hyun
Copy link
Member Author

Thank you, @peter-toth ! Merged to master.

@dongjoon-hyun dongjoon-hyun deleted the SPARK-55757 branch February 28, 2026 21:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants