[SPARK-43342][K8S] Revert SPARK-39006 Show a directional error message for executor PVC dynamic allocation failure

[dongjoon-hyun]

What changes were proposed in this pull request?

This reverts commit b065c94.

Why are the changes needed?

To remove the regression from SPARK-39006.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Pass the CIs.

Closes #41057

[dongjoon-hyun]

cc @dcoliversun This will be merged with your authorship.

[dongjoon-hyun]

After merging this clean-revert, you may want to make a new and proper test PR for https://github.com/apache/spark/pull/41057/files#r1186469992 if you want.

[dcoliversun]

@dongjoon-hyun Thanks for your PR. I'm +1

[dongjoon-hyun]

Thank you, @dcoliversun .

[dongjoon-hyun]

Could you review this PR, @viirya ?

[viirya]

May I ask why this is reverted?

[dongjoon-hyun]

Sorry for missing context, @viirya .

SPARK-43329 and SPARK-43342 were filed as a regression at Apache Spark 3.4.0 and those issues have the error message, IllegalArgumentException: PVC ClaimName: a1pvc should contain OnDemand or SPARK_EXECUTOR_ID when requiring multiple executors.

I linked all the reported issues to the original JIRA which caused this regression.

• https://issues.apache.org/jira/browse/SPARK-39006 Show a directional error message for PVC Dynamic Allocation Failure

And, the following was the reverting PR from the author of SPARK-39006, @dcoliversun.

• [SPARK-43342][K8S] Support driver and executors to shared same Kubernetes PVC  #41057

In short, while adding a log message, we missed a corner case which one NFS-backed PVC is mounted by multiple executors. So, we decide to revert SPARK-39006 because it doesn't give actual benefit (except logging).

[viirya]

Thank you @dongjoon-hyun

[dongjoon-hyun]

Thank you!

[dongjoon-hyun]

Merged to master/3.4.