Skip to content

Commit

Permalink
[SPARK-34486][K8S] Upgrade kubernetes-client to 4.13.2
Browse files Browse the repository at this point in the history
### What changes were proposed in this pull request?

This PR aims to upgrade `kubernetes-client` library from 4.12.0 to 4.13.2 for Apache Spark 3.2.0.

### Why are the changes needed?

This will bring [K8s 1.19.1](fabric8io/kubernetes-client#2541) models officially and the latest bug fixes.

- https://github.com/fabric8io/kubernetes-client/releases/tag/v4.13.0
- https://github.com/fabric8io/kubernetes-client/releases/tag/v4.13.1
- https://github.com/fabric8io/kubernetes-client/releases/tag/v4.13.2

### Does this PR introduce _any_ user-facing change?

No

### How was this patch tested?

Pass the K8s IT and UT.

```
KubernetesSuite:
- Run SparkPi with no resources
- Run SparkPi with a very long application name.
- Use SparkLauncher.NO_RESOURCE
- Run SparkPi with a master URL without a scheme.
- Run SparkPi with an argument.
- Run SparkPi with custom labels, annotations, and environment variables.
- All pods have the same service account by default
- Run extraJVMOptions check on driver
- Run SparkRemoteFileTest using a remote data file
- Verify logging configuration is picked from the provided SPARK_CONF_DIR/log4j.properties
- Run SparkPi with env and mount secrets.
- Run PySpark on simple pi.py example
- Run PySpark to test a pyfiles example
- Run PySpark with memory customization
- Run in client mode.
- Start pod creation from template
- PVs with local storage
- Launcher client dependencies
- SPARK-33615: Launcher client archives
- SPARK-33748: Launcher python client respecting PYSPARK_PYTHON
- SPARK-33748: Launcher python client respecting spark.pyspark.python and spark.pyspark.driver.python
- Launcher python client dependencies using a zip file
- Test basic decommissioning
- Test basic decommissioning with shuffle cleanup
- Test decommissioning with dynamic allocation & shuffle cleanups
- Test decommissioning timeouts
- Run SparkR on simple dataframe.R example
Run completed in 19 minutes, 25 seconds.
Total number of tests run: 27
Suites: completed 2, aborted 0
Tests: succeeded 27, failed 0, canceled 0, ignored 0, pending 0
All tests passed.
```

Closes #31602 from dongjoon-hyun/SPARK-34486.

Authored-by: Dongjoon Hyun <dhyun@apple.com>
Signed-off-by: HyukjinKwon <gurwls223@apache.org>
  • Loading branch information
dongjoon-hyun authored and HyukjinKwon committed Feb 21, 2021
1 parent 546d2eb commit 020e84e
Show file tree
Hide file tree
Showing 4 changed files with 44 additions and 42 deletions.
41 changes: 21 additions & 20 deletions dev/deps/spark-deps-hadoop-2.7-hive-2.3
Expand Up @@ -156,26 +156,27 @@ jsr305/3.0.0//jsr305-3.0.0.jar
jta/1.1//jta-1.1.jar
jul-to-slf4j/1.7.30//jul-to-slf4j-1.7.30.jar
kryo-shaded/4.0.2//kryo-shaded-4.0.2.jar
kubernetes-client/4.12.0//kubernetes-client-4.12.0.jar
kubernetes-model-admissionregistration/4.12.0//kubernetes-model-admissionregistration-4.12.0.jar
kubernetes-model-apiextensions/4.12.0//kubernetes-model-apiextensions-4.12.0.jar
kubernetes-model-apps/4.12.0//kubernetes-model-apps-4.12.0.jar
kubernetes-model-autoscaling/4.12.0//kubernetes-model-autoscaling-4.12.0.jar
kubernetes-model-batch/4.12.0//kubernetes-model-batch-4.12.0.jar
kubernetes-model-certificates/4.12.0//kubernetes-model-certificates-4.12.0.jar
kubernetes-model-common/4.12.0//kubernetes-model-common-4.12.0.jar
kubernetes-model-coordination/4.12.0//kubernetes-model-coordination-4.12.0.jar
kubernetes-model-core/4.12.0//kubernetes-model-core-4.12.0.jar
kubernetes-model-discovery/4.12.0//kubernetes-model-discovery-4.12.0.jar
kubernetes-model-events/4.12.0//kubernetes-model-events-4.12.0.jar
kubernetes-model-extensions/4.12.0//kubernetes-model-extensions-4.12.0.jar
kubernetes-model-metrics/4.12.0//kubernetes-model-metrics-4.12.0.jar
kubernetes-model-networking/4.12.0//kubernetes-model-networking-4.12.0.jar
kubernetes-model-policy/4.12.0//kubernetes-model-policy-4.12.0.jar
kubernetes-model-rbac/4.12.0//kubernetes-model-rbac-4.12.0.jar
kubernetes-model-scheduling/4.12.0//kubernetes-model-scheduling-4.12.0.jar
kubernetes-model-settings/4.12.0//kubernetes-model-settings-4.12.0.jar
kubernetes-model-storageclass/4.12.0//kubernetes-model-storageclass-4.12.0.jar
kubernetes-client/4.13.2//kubernetes-client-4.13.2.jar
kubernetes-model-admissionregistration/4.13.2//kubernetes-model-admissionregistration-4.13.2.jar
kubernetes-model-apiextensions/4.13.2//kubernetes-model-apiextensions-4.13.2.jar
kubernetes-model-apps/4.13.2//kubernetes-model-apps-4.13.2.jar
kubernetes-model-autoscaling/4.13.2//kubernetes-model-autoscaling-4.13.2.jar
kubernetes-model-batch/4.13.2//kubernetes-model-batch-4.13.2.jar
kubernetes-model-certificates/4.13.2//kubernetes-model-certificates-4.13.2.jar
kubernetes-model-common/4.13.2//kubernetes-model-common-4.13.2.jar
kubernetes-model-coordination/4.13.2//kubernetes-model-coordination-4.13.2.jar
kubernetes-model-core/4.13.2//kubernetes-model-core-4.13.2.jar
kubernetes-model-discovery/4.13.2//kubernetes-model-discovery-4.13.2.jar
kubernetes-model-events/4.13.2//kubernetes-model-events-4.13.2.jar
kubernetes-model-extensions/4.13.2//kubernetes-model-extensions-4.13.2.jar
kubernetes-model-metrics/4.13.2//kubernetes-model-metrics-4.13.2.jar
kubernetes-model-networking/4.13.2//kubernetes-model-networking-4.13.2.jar
kubernetes-model-node/4.13.2//kubernetes-model-node-4.13.2.jar
kubernetes-model-policy/4.13.2//kubernetes-model-policy-4.13.2.jar
kubernetes-model-rbac/4.13.2//kubernetes-model-rbac-4.13.2.jar
kubernetes-model-scheduling/4.13.2//kubernetes-model-scheduling-4.13.2.jar
kubernetes-model-settings/4.13.2//kubernetes-model-settings-4.13.2.jar
kubernetes-model-storageclass/4.13.2//kubernetes-model-storageclass-4.13.2.jar
leveldbjni-all/1.8//leveldbjni-all-1.8.jar
libfb303/0.9.3//libfb303-0.9.3.jar
libthrift/0.12.0//libthrift-0.12.0.jar
Expand Down
41 changes: 21 additions & 20 deletions dev/deps/spark-deps-hadoop-3.2-hive-2.3
Expand Up @@ -126,26 +126,27 @@ jsr305/3.0.0//jsr305-3.0.0.jar
jta/1.1//jta-1.1.jar
jul-to-slf4j/1.7.30//jul-to-slf4j-1.7.30.jar
kryo-shaded/4.0.2//kryo-shaded-4.0.2.jar
kubernetes-client/4.12.0//kubernetes-client-4.12.0.jar
kubernetes-model-admissionregistration/4.12.0//kubernetes-model-admissionregistration-4.12.0.jar
kubernetes-model-apiextensions/4.12.0//kubernetes-model-apiextensions-4.12.0.jar
kubernetes-model-apps/4.12.0//kubernetes-model-apps-4.12.0.jar
kubernetes-model-autoscaling/4.12.0//kubernetes-model-autoscaling-4.12.0.jar
kubernetes-model-batch/4.12.0//kubernetes-model-batch-4.12.0.jar
kubernetes-model-certificates/4.12.0//kubernetes-model-certificates-4.12.0.jar
kubernetes-model-common/4.12.0//kubernetes-model-common-4.12.0.jar
kubernetes-model-coordination/4.12.0//kubernetes-model-coordination-4.12.0.jar
kubernetes-model-core/4.12.0//kubernetes-model-core-4.12.0.jar
kubernetes-model-discovery/4.12.0//kubernetes-model-discovery-4.12.0.jar
kubernetes-model-events/4.12.0//kubernetes-model-events-4.12.0.jar
kubernetes-model-extensions/4.12.0//kubernetes-model-extensions-4.12.0.jar
kubernetes-model-metrics/4.12.0//kubernetes-model-metrics-4.12.0.jar
kubernetes-model-networking/4.12.0//kubernetes-model-networking-4.12.0.jar
kubernetes-model-policy/4.12.0//kubernetes-model-policy-4.12.0.jar
kubernetes-model-rbac/4.12.0//kubernetes-model-rbac-4.12.0.jar
kubernetes-model-scheduling/4.12.0//kubernetes-model-scheduling-4.12.0.jar
kubernetes-model-settings/4.12.0//kubernetes-model-settings-4.12.0.jar
kubernetes-model-storageclass/4.12.0//kubernetes-model-storageclass-4.12.0.jar
kubernetes-client/4.13.2//kubernetes-client-4.13.2.jar
kubernetes-model-admissionregistration/4.13.2//kubernetes-model-admissionregistration-4.13.2.jar
kubernetes-model-apiextensions/4.13.2//kubernetes-model-apiextensions-4.13.2.jar
kubernetes-model-apps/4.13.2//kubernetes-model-apps-4.13.2.jar
kubernetes-model-autoscaling/4.13.2//kubernetes-model-autoscaling-4.13.2.jar
kubernetes-model-batch/4.13.2//kubernetes-model-batch-4.13.2.jar
kubernetes-model-certificates/4.13.2//kubernetes-model-certificates-4.13.2.jar
kubernetes-model-common/4.13.2//kubernetes-model-common-4.13.2.jar
kubernetes-model-coordination/4.13.2//kubernetes-model-coordination-4.13.2.jar
kubernetes-model-core/4.13.2//kubernetes-model-core-4.13.2.jar
kubernetes-model-discovery/4.13.2//kubernetes-model-discovery-4.13.2.jar
kubernetes-model-events/4.13.2//kubernetes-model-events-4.13.2.jar
kubernetes-model-extensions/4.13.2//kubernetes-model-extensions-4.13.2.jar
kubernetes-model-metrics/4.13.2//kubernetes-model-metrics-4.13.2.jar
kubernetes-model-networking/4.13.2//kubernetes-model-networking-4.13.2.jar
kubernetes-model-node/4.13.2//kubernetes-model-node-4.13.2.jar
kubernetes-model-policy/4.13.2//kubernetes-model-policy-4.13.2.jar
kubernetes-model-rbac/4.13.2//kubernetes-model-rbac-4.13.2.jar
kubernetes-model-scheduling/4.13.2//kubernetes-model-scheduling-4.13.2.jar
kubernetes-model-settings/4.13.2//kubernetes-model-settings-4.13.2.jar
kubernetes-model-storageclass/4.13.2//kubernetes-model-storageclass-4.13.2.jar
leveldbjni-all/1.8//leveldbjni-all-1.8.jar
libfb303/0.9.3//libfb303-0.9.3.jar
libthrift/0.12.0//libthrift-0.12.0.jar
Expand Down
2 changes: 1 addition & 1 deletion resource-managers/kubernetes/core/pom.xml
Expand Up @@ -30,7 +30,7 @@
<properties>
<sbt.project.name>kubernetes</sbt.project.name>
<!-- Note: Please update the kubernetes client version in kubernetes/integration-tests/pom.xml -->
<kubernetes.client.version>4.12.0</kubernetes.client.version>
<kubernetes.client.version>4.13.2</kubernetes.client.version>
</properties>

<dependencies>
Expand Down
2 changes: 1 addition & 1 deletion resource-managers/kubernetes/integration-tests/pom.xml
Expand Up @@ -28,7 +28,7 @@
<properties>
<download-maven-plugin.version>1.3.0</download-maven-plugin.version>
<extraScalaTestArgs></extraScalaTestArgs>
<kubernetes-client.version>4.12.0</kubernetes-client.version>
<kubernetes-client.version>4.13.2</kubernetes-client.version>
<sbt.project.name>kubernetes-integration-tests</sbt.project.name>

<!-- Integration Test Configuration Properties -->
Expand Down

0 comments on commit 020e84e

Please sign in to comment.