[SPARK-38048][K8S][TESTS] Add IntegrationTestBackend.describePods to support all K8s test backends

[williamhyun]

What changes were proposed in this pull request?

This PR aims to add IntegrationTestBackend.describePods to support all K8s test backends

Why are the changes needed?

Currently the docker based K8s tests cannot get the pod information when it fails.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Manually.

[dongjoon-hyun]

Nice. Thank you, @williamhyun .
+1, LGTM. Merged to master.

[dongjoon-hyun]

I tested and backported this test patch to branch-3.2 for Apache Spark 3.2.2. Thanks, @williamhyun .

[info] KubernetesSuite:
[info] - Run SparkPi with no resources (8 seconds, 818 milliseconds)
[info] - Run SparkPi with a very long application name. (8 seconds, 413 milliseconds)
[info] - Use SparkLauncher.NO_RESOURCE (8 seconds, 364 milliseconds)
[info] - Run SparkPi with a master URL without a scheme. (8 seconds, 403 milliseconds)
[info] - Run SparkPi with an argument. (8 seconds, 263 milliseconds)
[info] - Run SparkPi with custom labels, annotations, and environment variables. (8 seconds, 296 milliseconds)
[info] - All pods have the same service account by default (8 seconds, 315 milliseconds)
[info] - Run extraJVMOptions check on driver (4 seconds, 145 milliseconds)
[info] - Run SparkRemoteFileTest using a remote data file (8 seconds, 236 milliseconds)
[info] - Verify logging configuration is picked from the provided SPARK_CONF_DIR/log4j.properties (8 seconds, 562 milliseconds)
[info] - Run SparkPi with env and mount secrets. (18 seconds, 2 milliseconds)
[info] - Run PySpark on simple pi.py example (10 seconds, 340 milliseconds)
[info] - Run PySpark to test a pyfiles example (10 seconds, 426 milliseconds)
[info] - Run PySpark with memory customization (9 seconds, 350 milliseconds)
[info] - Run in client mode. (7 seconds, 368 milliseconds)
[info] - Start pod creation from template (8 seconds, 426 milliseconds)
[info] - Test basic decommissioning (42 seconds, 223 milliseconds)
[info] - Test basic decommissioning with shuffle cleanup (44 seconds, 154 milliseconds)
[info] - Test decommissioning with dynamic allocation & shuffle cleanups (2 minutes, 40 seconds)
[info] - Test decommissioning timeouts (42 seconds, 165 milliseconds)
[info] - Run SparkR on simple dataframe.R example (11 seconds, 406 milliseconds)
[info] Run completed in 9 minutes, 33 seconds.
[info] Total number of tests run: 21
[info] Suites: completed 1, aborted 0
[info] Tests: succeeded 21, failed 0, canceled 0, ignored 0, pending 0
[info] All tests passed.
[success] Total time: 715 s (11:55), completed Mar 2, 2022, 11:37:15 AM