[SPARK-44442][MESOS] Remove Mesos support

[srowen]

What changes were proposed in this pull request?

Remove Mesos resource manager support and docs, previously deprecated in Spark 3.2, for Spark 4.0.

Why are the changes needed?

Mesos is no longer supported and has been deprecated for several versions.

Does this PR introduce any user-facing change?

Other than the lack of Mesos support, no

How was this patch tested?

Existing tests.

Was this patch authored or co-authored using generative AI tooling?

No

[mridulm]

9 lines more, and it would have been an event 10k lines removed :-)

[LuciferYang]

9 lines more, and it would have been an event 10k lines removed :-)

A satisfying operation, hahaha ~

[LuciferYang]

spark/core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala

Lines 814 to 831 in f6c6acb

	if (state == TaskState.LOST) {
	// TaskState.LOST is only used by the deprecated Mesos fine-grained scheduling mode,
	// where each executor corresponds to a single task, so mark the executor as failed.
	val execId = taskIdToExecutorId.getOrElse(tid, {
	val errorMsg =
	"taskIdToTaskSetManager.contains(tid) <=> taskIdToExecutorId.contains(tid)"
	taskSet.abort(errorMsg)
	throw new SparkException(
	"taskIdToTaskSetManager.contains(tid) <=> taskIdToExecutorId.contains(tid)")
	})
	if (executorIdToRunningTaskIds.contains(execId)) {
	reason = Some(
	ExecutorProcessLost(
	s"Task $tid was lost, so marking the executor as lost as well."))
	removeExecutor(execId, reason.get)
	failedExecutor = Some(execId)
	}
	}

Does this part need to be fixed?

[LuciferYang]

spark/core/src/main/scala/org/apache/spark/SparkContext.scala

Line 2728 in f6c6acb

* The reasons for this are discussed in https://github.com/mesos/spark/pull/718

In addition, does this line of comment still need to be retained?

[srowen]

I noticed the TaskState.LOST comment, but was reluctant to actually totally remove handling of this event in case the comment was wrong.
the link to the explanation in the Mesos issue is still valid, just happens to come from a really old Mesos issue on Github

[LuciferYang]

I noticed the TaskState.LOST comment, but was reluctant to actually totally remove handling of this event in case the comment was wrong. the link to the explanation in the Mesos issue is still valid, just happens to come from a really old Mesos issue on Github

fine to me

[srowen]

It looks like the tests actually passed but showing failed - I'm not quite sure what to make of the failure in 'reporting results'.
EDIT: Oh wait now it shows passed.

[srowen]

Merged to master

[dongjoon-hyun]

+1, late LGTM.