[SPARK-56684][K8S] Expose KubernetesClusterSchedulerBackend.kubernetesClient to k8s package

[dongjoon-hyun]

What changes were proposed in this pull request?

This PR promotes the kubernetesClient constructor parameter of KubernetesClusterSchedulerBackend to a private[k8s] val, exposing it as a read-only member to other components in the org.apache.spark.scheduler.cluster.k8s package.

Why are the changes needed?

Like ExecutorResizePlugin, many plugins currently build its own KubernetesClient via SparkKubernetesClientFactory.createKubernetesClient(...). This produces multiple clients in the driver process that target the same API server with the same credentials.

The driver already owns a single KubernetesClient through KubernetesClusterSchedulerBackend, whose lifecycle is managed in start() / stop(). Exposing it as private[k8s] val lets package-local components reuse this shared client in follow-up work, without widening the API surface to other modules or external users.

This PR is intentionally limited to the visibility change. Reusing the shared client from existing plugins is left to follow-up PRs so each change stays minimal and reviewable.

Does this PR introduce any user-facing change?

No. The visibility is private[k8s], so the change is invisible outside the org.apache.spark.scheduler.cluster.k8s package.

How was this patch tested?

Pass the CIs with a new unit test.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Opus 4.7

[dongjoon-hyun]

Could you review this K8s API visibility PR, @huaxingao ?

This is a subtask of SPARK-56603 Improve K8s Resource Manager API

[huaxingao]

LGTM

[dongjoon-hyun]

Thank you so much, @huaxingao ~

[dongjoon-hyun]

Merged to master for Apache Spark 4.2.0.