[SPARK-36026][BUILD][K8S] Upgrade kubernetes-client to 5.5.0

[attilapiros]

What changes were proposed in this pull request?

Upgrading the kubernetes-client to 5.5.0

Why are the changes needed?

There are several bugfixes but the main reason is version 5.5.0 contains Support HTTP operation retry with exponential backoff (for status code >= 500).

Does this PR introduce any user-facing change?

No.

How was this patch tested?

By running the integration tests including persistentVolume tests:

./resource-managers/kubernetes/integration-tests/dev/dev-run-integration-tests.sh \
    --spark-tgz $TARBALL_TO_TEST --hadoop-profile $HADOOP_PROFILE --exclude-tags r --include-tags persistentVolume
...
[INFO] --- scalatest-maven-plugin:2.0.0:test (integration-test) @ spark-kubernetes-integration-tests_2.12 ---
Discovery starting.
Discovery completed in 413 milliseconds.
Run starting. Expected test count is: 26
KubernetesSuite:
- Run SparkPi with no resources
- Run SparkPi with a very long application name.
- Use SparkLauncher.NO_RESOURCE
- Run SparkPi with a master URL without a scheme.
- Run SparkPi with an argument.
- Run SparkPi with custom labels, annotations, and environment variables.
- All pods have the same service account by default
- Run extraJVMOptions check on driver
- Run SparkRemoteFileTest using a remote data file
- Verify logging configuration is picked from the provided SPARK_CONF_DIR/log4j.properties
- Run SparkPi with env and mount secrets.
- Run PySpark on simple pi.py example
- Run PySpark to test a pyfiles example
- Run PySpark with memory customization
- Run in client mode.
- Start pod creation from template
- PVs with local storage
- Launcher client dependencies
- SPARK-33615: Launcher client archives
- SPARK-33748: Launcher python client respecting PYSPARK_PYTHON
- SPARK-33748: Launcher python client respecting spark.pyspark.python and spark.pyspark.driver.python
- Launcher python client dependencies using a zip file
- Test basic decommissioning
- Test basic decommissioning with shuffle cleanup
- Test decommissioning with dynamic allocation & shuffle cleanups
- Test decommissioning timeouts
Run completed in 18 minutes, 34 seconds.
Total number of tests run: 26
Suites: completed 2, aborted 0
Tests: succeeded 26, failed 0, canceled 0, ignored 0, pending 0
All tests passed.

Checked the compatibility matrix and the same k8s versions are supported as were by version 5.4.1.

[attilapiros]

cc @dongjoon-hyun @srowen

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45219/

[SparkQA]

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45219/

[SparkQA]

Test build #140708 has finished for PR 33233 at commit fb3054a.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[HyukjinKwon]

Merged to master.

[dongjoon-hyun]

+1, late LGTM. Thanks, @attilapiros and @HyukjinKwon

[dongjoon-hyun]

For the following bug fix, could you make a backporting PR please, @attilapiros ?

There are several bugfixes but the main reason is version 5.5.0 contains Support HTTP operation retry with exponential backoff (for status code >= 500).

[attilapiros]

@dongjoon-hyun

Sure, which branches do you have in mind?

For the mentioned kubernetes-client feature we will need this one too: #33261
Otherwise only a few people will profit about having this available in Spark.

[dongjoon-hyun]

If you think so, never mind, @attilapiros .