[SPARK-38652][K8S] uploadFileUri should preserve file scheme

[dongjoon-hyun]

What changes were proposed in this pull request?

This PR replaces new Path(fileUri.getPath) with new Path(fileUri).
By using Path class constructor with URI parameter, we can preserve file scheme.

Why are the changes needed?

If we use, Path class constructor with String parameter, it loses file scheme information.
Although the original code works so far, it fails at Apache Hadoop 3.3.2 and breaks dependency upload feature which is covered by K8s Minikube integration tests.

test("uploadFileUri") {
   val fileUri = org.apache.spark.util.Utils.resolveURI("/tmp/1.txt")
   assert(new Path(fileUri).toString == "file:/private/tmp/1.txt")
   assert(new Path(fileUri.getPath).toString == "/private/tmp/1.txt")
}

Does this PR introduce any user-facing change?

No, this will prevent a regression at Apache Spark 3.3.0 instead.

How was this patch tested?

Pass the CIs.

In addition, this PR and #36009 will recover K8s IT DepsTestsSuite.

[LuciferYang]

LGTM

[dcoliversun]

+1, LGTM.

[dcoliversun]

@dongjoon-hyun I agree this PR could fix SPARK-38652.
Does spark upload file with schema file:// using hadoop-aws-3.3.2 still throw a PathIOException in normal usage job other than DepsTestsSuite?

[dongjoon-hyun]

Thank you, @LuciferYang , @dcoliversun , @wangyum . Merged to master/3.3/3.2/3.1/3.0.

All tests including K8s IT passed.

KubernetesSuite:
- Run SparkPi with no resources
- Run SparkPi with no resources & statefulset allocation
- Run SparkPi with a very long application name.
- Use SparkLauncher.NO_RESOURCE
- Run SparkPi with a master URL without a scheme.
- Run SparkPi with an argument.
- Run SparkPi with custom labels, annotations, and environment variables.
- All pods have the same service account by default
- Run extraJVMOptions check on driver
- Run SparkRemoteFileTest using a remote data file
- Verify logging configuration is picked from the provided SPARK_CONF_DIR/log4j2.properties
- Run SparkPi with env and mount secrets.
- Run PySpark on simple pi.py example
- Run PySpark to test a pyfiles example
- Run PySpark with memory customization
- Run in client mode.
- Start pod creation from template
- SPARK-38398: Schedule pod creation from template
- PVs with local hostpath storage on statefulsets
- PVs with local hostpath and storageClass on statefulsets
- PVs with local storage
- Launcher client dependencies
- SPARK-33615: Launcher client archives
- SPARK-33748: Launcher python client respecting PYSPARK_PYTHON
- SPARK-33748: Launcher python client respecting spark.pyspark.python and spark.pyspark.driver.python
- Launcher python client dependencies using a zip file
- Test basic decommissioning
- Test basic decommissioning with shuffle cleanup
- Test decommissioning with dynamic allocation & shuffle cleanups
- Test decommissioning timeouts
- SPARK-37576: Rolling decommissioning
Run completed in 22 minutes, 16 seconds.
Total number of tests run: 31
Suites: completed 2, aborted 0
Tests: succeeded 31, failed 0, canceled 0, ignored 0, pending 0
All tests passed.

To @dcoliversun . No, it should not. This PR aims to fix uploadFileUri itself for all usages. Like the test case written in the PR description, Apache Spark uploadFileUri uses Utils.resolveURI(uri) to add file scheme before invoking uploadFileToHadoopCompatibleFS.

spark/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/KubernetesUtils.scala

Line 314 in 6b29b28

val fileUri = Utils.resolveURI(uri)