[SPARK-59124][CORE][K8S] Support spark.kubernetes.files.avoidDownloadSchemes for K8s Cluster Mode

[WonYong-Jang]

What changes were proposed in this pull request?

Add a new config spark.kubernetes.files.avoidDownloadSchemes that extends the existing avoidDownload mechanism (SPARK-47475, for jars) to spark.files in Kubernetes cluster mode. When a file's scheme matches (or *), its remote URI is preserved instead of being downloaded to the driver's local disk. This mirrors the jars handling.

Why are the changes needed?

In K8s cluster mode, SparkSubmit always downloads spark.files to the driver and rewrites URIs to file:/tmp/..., discarding the original remote URI (e.g. s3a://). As a result spark.read.* cannot resolve the file on executors, and all files funnel through the driver's file server, saturating its network at high executor counts. Jars already avoid this; files should have the same option.

Does this PR introduce any user-facing change?

Yes - a new opt-in config spark.kubernetes.files.avoidDownloadSchemes (default: none)
No behavior change unless set.

How was this patch tested?

New SparkSubmitSuite tests covering scheme-match(s3a), wildcard (*), and non-match(hdfs) cases.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Code (Claude Opus 4.8)

Closes #58310

[WonYong-Jang]

@dongjoon-hyun
This extends the same avoidDownloadSchemes option to spark.files, following
SPARK-47475 (jars) and SPARK-55077. Would appreciate a review when you have time.