[SPARK-46888][CORE] Fix Master to reject /workers/kill/ requests if decommission is disabled

[dongjoon-hyun]

What changes were proposed in this pull request?

This PR aims to fix Master to reject /workers/kill/ request if spark.decommission.enabled is false in order to fix the dangling worker issue.

Why are the changes needed?

Currently, spark.decommission.enabled is false by default. So, when a user asks to decommission, only Master marked it DECOMMISSIONED while the worker is alive.

$ curl -XPOST http://localhost:8080/workers/kill/\?host\=127.0.0.1

Master UI

Worker Log

24/01/27 18:18:06 WARN Worker: Receive decommission request, but decommission feature is disabled.

To be consistent with the existing Worker behavior which ignores the request.

spark/core/src/main/scala/org/apache/spark/deploy/worker/Worker.scala

Lines 859 to 868 in 1787a52

	private[deploy] def decommissionSelf(): Unit = {
	if (conf.get(config.DECOMMISSION_ENABLED) && !decommissioned) {
	decommissioned = true
	logInfo(s"Decommission worker $workerId.")
	} else if (decommissioned) {
	logWarning(s"Worker $workerId already started decommissioning.")
	} else {
	logWarning(s"Receive decommission request, but decommission feature is disabled.")
	}
	}

Does this PR introduce any user-facing change?

No, this is a bug fix.

How was this patch tested?

Pass the CI with the newly added test case.

Was this patch authored or co-authored using generative AI tooling?

No.

[dongjoon-hyun]

Also, could you review this dangled Workers bug @viirya ?

[dongjoon-hyun]

Oh, wait. I'm revising the test case.

[dongjoon-hyun]

Thank you so much again, @viirya .

[dongjoon-hyun]

Merged to master/3.5/3.4.