[SPARK-55200][K8S][TESTS] Add RancherDesktopBackend for K8s integration tests

[dongjoon-hyun]

What changes were proposed in this pull request?

This PR aims to add RancherDesktopBackend for K8s integration tests in addition to the existing minikube, docker-desktop, and cloud-backend. Like the other backends, this PR doesn't aim to make all test cases passes. In other words, minikube (the default backend) is the only test backend for all test coverage.

Why are the changes needed?

Rancher Desktop is another open-source (Apache 2.0 license) project that brings Kubernetes and container management to the desktop. It runs on Windows, macOS and Linux. Unlike Docker Desktop, this is available freely for all purposes.

It would be great if we can support Rancher Desktop as an alternative developer tools.

Does this PR introduce any user-facing change?

No behavior change.

How was this patch tested?

Pass the CIs and manual tests.

$ NO_MANUAL=1 ./dev/make-distribution.sh --pip --tgz -Pkubernetes,hadoop-3,hadoop-cloud,hive

$ resource-managers/kubernetes/integration-tests/dev/dev-run-integration-tests.sh \
--deploy-mode rancher-desktop --exclude-tags minikube,r,local,command --spark-tgz $PWD/spark-*.tgz
...
VolcanoSuite:
- SPARK-42190: Run SparkPi with local[*]
- Run SparkPi with no resources
- SPARK-53944: Run SparkPi without driver service
- Run SparkPi with no resources & statefulset allocation
- Run SparkPi with a very long application name.
- Use SparkLauncher.NO_RESOURCE
- Run SparkPi with a master URL without a scheme.
- Run SparkPi with an argument.
- Run SparkPi with custom labels, annotations, and environment variables.
- All pods have the same service account by default
- Run extraJVMOptions check on driver
- SPARK-42474: Run extraJVMOptions JVM GC option check - G1GC
- SPARK-42474: Run extraJVMOptions JVM GC option check - Other GC
- SPARK-42769: All executor pods have SPARK_DRIVER_POD_IP env variable
- Verify logging configuration is picked from the provided SPARK_CONF_DIR/log4j2.properties
- Run PySpark on simple pi.py example
- Run PySpark to test a pyfiles example
- Run PySpark with memory customization
- Run PySpark with Spark Connect !!! IGNORED !!!
- Run in client mode.
- Start pod creation from template
- SPARK-38398: Schedule pod creation from template
- PVs with local hostpath storage on statefulsets !!! CANCELED !!!
  "[Volcano]Suite" did not equal "[Kubernetes]Suite" (PVTestsSuite.scala:116)
- Test basic decommissioning
- Test basic decommissioning with shuffle cleanup
- Test decommissioning with dynamic allocation & shuffle cleanups
- Test decommissioning timeouts
- SPARK-37576: Rolling decommissioning
- Run SparkPi with volcano scheduler
- SPARK-38187: Run SparkPi Jobs with minCPU
- SPARK-38187: Run SparkPi Jobs with minMemory
- SPARK-38188: Run SparkPi jobs with 2 queues (only 1 enabled)
- SPARK-38188: Run SparkPi jobs with 2 queues (all enabled)
- SPARK-38423: Run driver job to validate priority order
KubernetesSuite:
- SPARK-42190: Run SparkPi with local[*]
- Run SparkPi with no resources
- SPARK-53944: Run SparkPi without driver service
- Run SparkPi with no resources & statefulset allocation
- Run SparkPi with a very long application name.
- Use SparkLauncher.NO_RESOURCE
- Run SparkPi with a master URL without a scheme.
- Run SparkPi with an argument.
- Run SparkPi with custom labels, annotations, and environment variables.
- All pods have the same service account by default
- Run extraJVMOptions check on driver
- SPARK-42474: Run extraJVMOptions JVM GC option check - G1GC
- SPARK-42474: Run extraJVMOptions JVM GC option check - Other GC
- SPARK-42769: All executor pods have SPARK_DRIVER_POD_IP env variable
- Verify logging configuration is picked from the provided SPARK_CONF_DIR/log4j2.properties
- Run PySpark on simple pi.py example
- Run PySpark to test a pyfiles example
- Run PySpark with memory customization
- Run PySpark with Spark Connect !!! IGNORED !!!
- Run in client mode.
- Start pod creation from template
- SPARK-38398: Schedule pod creation from template
- PVs with local hostpath storage on statefulsets *** FAILED ***
- Test basic decommissioning
- Test basic decommissioning with shuffle cleanup
- Test decommissioning with dynamic allocation & shuffle cleanups
- Test decommissioning timeouts
- SPARK-37576: Rolling decommissioning
YuniKornSuite:
Run completed in 28 minutes, 13 seconds.
Total number of tests run: 59
Suites: completed 4, aborted 0
Tests: succeeded 58, failed 1, canceled 1, ignored 2, pending 0

This PR intentionally show the above result including one test case failure which we can simply exclude it by --exclude-tags pv. Currently, the above example used --exclude-tags minikube,r,local,command. docker-desktop and cloud backends also have a limited test coverage and needs similar exclusion because some test cases are written for specific backends.

Was this patch authored or co-authored using generative AI tooling?

No.

[github-actions]

JIRA Issue Information

=== Sub-task SPARK-55200 ===
Summary: Add RancherDesktopBackend for K8s integration tests
Assignee: None
Status: Open
Affected: ["4.2.0"]

---

This comment was automatically generated by GitHub Actions

[dongjoon-hyun]

Could you review this test PR when you have some time, @peter-toth ?

As I described in the PR description, I've verified this PR with RancherDesktop.

[dongjoon-hyun]

Thank you so much, @peter-toth !

[dongjoon-hyun]

Merged to master for Apache Spark 4.2.0.