Skip to content

Commit

Permalink
[SPARK-31786][K8S][BUILD] Upgrade kubernetes-client to 4.9.2
Browse files Browse the repository at this point in the history
### What changes were proposed in this pull request?

This PR aims to upgrade `kubernetes-client` library to bring the JDK8 related fixes. Please note that JDK11 works fine without any problem.
- https://github.com/fabric8io/kubernetes-client/releases/tag/v4.9.2
  - JDK8 always uses http/1.1 protocol (Prevent OkHttp from wrongly enabling http/2)

### Why are the changes needed?

OkHttp "wrongly" detects the Platform as Jdk9Platform on JDK 8u251.
- fabric8io/kubernetes-client#2212
- https://stackoverflow.com/questions/61565751/why-am-i-not-able-to-run-sparkpi-example-on-a-kubernetes-k8s-cluster

Although there is a workaround `export HTTP2_DISABLE=true` and `Downgrade JDK or K8s`, we had better avoid this problematic situation.

### Does this PR introduce _any_ user-facing change?

No. This will recover the failures on JDK 8u252.

### How was this patch tested?

- [x] Pass the Jenkins UT (#28601 (comment))
- [x] Pass the Jenkins K8S IT with the K8s 1.13 (#28601 (comment))
- [x] Manual testing with K8s 1.17.3. (Below)

**v1.17.6 result (on Minikube)**
```
KubernetesSuite:
- Run SparkPi with no resources
- Run SparkPi with a very long application name.
- Use SparkLauncher.NO_RESOURCE
- Run SparkPi with a master URL without a scheme.
- Run SparkPi with an argument.
- Run SparkPi with custom labels, annotations, and environment variables.
- All pods have the same service account by default
- Run extraJVMOptions check on driver
- Run SparkRemoteFileTest using a remote data file
- Run SparkPi with env and mount secrets.
- Run PySpark on simple pi.py example
- Run PySpark with Python2 to test a pyfiles example
- Run PySpark with Python3 to test a pyfiles example
- Run PySpark with memory customization
- Run in client mode.
- Start pod creation from template
- PVs with local storage
- Launcher client dependencies
- Test basic decommissioning
Run completed in 8 minutes, 27 seconds.
Total number of tests run: 19
Suites: completed 2, aborted 0
Tests: succeeded 19, failed 0, canceled 0, ignored 0, pending 0
All tests passed.
```

Closes #28601 from dongjoon-hyun/SPARK-K8S-CLIENT.

Authored-by: Dongjoon Hyun <dongjoon@apache.org>
Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
(cherry picked from commit 64ffc66)
Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
  • Loading branch information
dongjoon-hyun committed May 24, 2020
1 parent 1e79c0d commit f05a26a
Show file tree
Hide file tree
Showing 5 changed files with 14 additions and 11 deletions.
7 changes: 4 additions & 3 deletions dev/deps/spark-deps-hadoop-2.7-hive-1.2
Expand Up @@ -93,6 +93,7 @@ jackson-core-asl/1.9.13//jackson-core-asl-1.9.13.jar
jackson-core/2.10.0//jackson-core-2.10.0.jar
jackson-databind/2.10.0//jackson-databind-2.10.0.jar
jackson-dataformat-yaml/2.10.0//jackson-dataformat-yaml-2.10.0.jar
jackson-datatype-jsr310/2.10.3//jackson-datatype-jsr310-2.10.3.jar
jackson-jaxrs/1.9.13//jackson-jaxrs-1.9.13.jar
jackson-mapper-asl/1.9.13//jackson-mapper-asl-1.9.13.jar
jackson-module-jaxb-annotations/2.10.0//jackson-module-jaxb-annotations-2.10.0.jar
Expand Down Expand Up @@ -137,9 +138,9 @@ jsr305/3.0.0//jsr305-3.0.0.jar
jta/1.1//jta-1.1.jar
jul-to-slf4j/1.7.30//jul-to-slf4j-1.7.30.jar
kryo-shaded/4.0.2//kryo-shaded-4.0.2.jar
kubernetes-client/4.7.1//kubernetes-client-4.7.1.jar
kubernetes-model-common/4.7.1//kubernetes-model-common-4.7.1.jar
kubernetes-model/4.7.1//kubernetes-model-4.7.1.jar
kubernetes-client/4.9.2//kubernetes-client-4.9.2.jar
kubernetes-model-common/4.9.2//kubernetes-model-common-4.9.2.jar
kubernetes-model/4.9.2//kubernetes-model-4.9.2.jar
leveldbjni-all/1.8//leveldbjni-all-1.8.jar
libfb303/0.9.3//libfb303-0.9.3.jar
libthrift/0.12.0//libthrift-0.12.0.jar
Expand Down
7 changes: 4 additions & 3 deletions dev/deps/spark-deps-hadoop-2.7-hive-2.3
Expand Up @@ -106,6 +106,7 @@ jackson-core-asl/1.9.13//jackson-core-asl-1.9.13.jar
jackson-core/2.10.0//jackson-core-2.10.0.jar
jackson-databind/2.10.0//jackson-databind-2.10.0.jar
jackson-dataformat-yaml/2.10.0//jackson-dataformat-yaml-2.10.0.jar
jackson-datatype-jsr310/2.10.3//jackson-datatype-jsr310-2.10.3.jar
jackson-jaxrs/1.9.13//jackson-jaxrs-1.9.13.jar
jackson-mapper-asl/1.9.13//jackson-mapper-asl-1.9.13.jar
jackson-module-jaxb-annotations/2.10.0//jackson-module-jaxb-annotations-2.10.0.jar
Expand Down Expand Up @@ -152,9 +153,9 @@ jsr305/3.0.0//jsr305-3.0.0.jar
jta/1.1//jta-1.1.jar
jul-to-slf4j/1.7.30//jul-to-slf4j-1.7.30.jar
kryo-shaded/4.0.2//kryo-shaded-4.0.2.jar
kubernetes-client/4.7.1//kubernetes-client-4.7.1.jar
kubernetes-model-common/4.7.1//kubernetes-model-common-4.7.1.jar
kubernetes-model/4.7.1//kubernetes-model-4.7.1.jar
kubernetes-client/4.9.2//kubernetes-client-4.9.2.jar
kubernetes-model-common/4.9.2//kubernetes-model-common-4.9.2.jar
kubernetes-model/4.9.2//kubernetes-model-4.9.2.jar
leveldbjni-all/1.8//leveldbjni-all-1.8.jar
libfb303/0.9.3//libfb303-0.9.3.jar
libthrift/0.12.0//libthrift-0.12.0.jar
Expand Down
7 changes: 4 additions & 3 deletions dev/deps/spark-deps-hadoop-3.2-hive-2.3
Expand Up @@ -105,6 +105,7 @@ jackson-core-asl/1.9.13//jackson-core-asl-1.9.13.jar
jackson-core/2.10.0//jackson-core-2.10.0.jar
jackson-databind/2.10.0//jackson-databind-2.10.0.jar
jackson-dataformat-yaml/2.10.0//jackson-dataformat-yaml-2.10.0.jar
jackson-datatype-jsr310/2.10.3//jackson-datatype-jsr310-2.10.3.jar
jackson-jaxrs-base/2.9.5//jackson-jaxrs-base-2.9.5.jar
jackson-jaxrs-json-provider/2.9.5//jackson-jaxrs-json-provider-2.9.5.jar
jackson-mapper-asl/1.9.13//jackson-mapper-asl-1.9.13.jar
Expand Down Expand Up @@ -164,9 +165,9 @@ kerby-pkix/1.0.1//kerby-pkix-1.0.1.jar
kerby-util/1.0.1//kerby-util-1.0.1.jar
kerby-xdr/1.0.1//kerby-xdr-1.0.1.jar
kryo-shaded/4.0.2//kryo-shaded-4.0.2.jar
kubernetes-client/4.7.1//kubernetes-client-4.7.1.jar
kubernetes-model-common/4.7.1//kubernetes-model-common-4.7.1.jar
kubernetes-model/4.7.1//kubernetes-model-4.7.1.jar
kubernetes-client/4.9.2//kubernetes-client-4.9.2.jar
kubernetes-model-common/4.9.2//kubernetes-model-common-4.9.2.jar
kubernetes-model/4.9.2//kubernetes-model-4.9.2.jar
leveldbjni-all/1.8//leveldbjni-all-1.8.jar
libfb303/0.9.3//libfb303-0.9.3.jar
libthrift/0.12.0//libthrift-0.12.0.jar
Expand Down
2 changes: 1 addition & 1 deletion resource-managers/kubernetes/core/pom.xml
Expand Up @@ -30,7 +30,7 @@
<properties>
<sbt.project.name>kubernetes</sbt.project.name>
<!-- Note: Please update the kubernetes client version in kubernetes/integration-tests/pom.xml -->
<kubernetes.client.version>4.7.1</kubernetes.client.version>
<kubernetes.client.version>4.9.2</kubernetes.client.version>
</properties>

<dependencies>
Expand Down
2 changes: 1 addition & 1 deletion resource-managers/kubernetes/integration-tests/pom.xml
Expand Up @@ -29,7 +29,7 @@
<download-maven-plugin.version>1.3.0</download-maven-plugin.version>
<exec-maven-plugin.version>1.4.0</exec-maven-plugin.version>
<extraScalaTestArgs></extraScalaTestArgs>
<kubernetes-client.version>4.7.1</kubernetes-client.version>
<kubernetes-client.version>4.9.2</kubernetes-client.version>
<scala-maven-plugin.version>3.2.2</scala-maven-plugin.version>
<scalatest-maven-plugin.version>1.0</scalatest-maven-plugin.version>
<sbt.project.name>kubernetes-integration-tests</sbt.project.name>
Expand Down

0 comments on commit f05a26a

Please sign in to comment.