[SPARK-38302][K8S][TESTS] Use `Java 17` in K8S IT in case of `spark-tgz` option #35627

dcoliversun · 2022-02-23T10:29:41Z

What changes were proposed in this pull request?

This PR aims to use Java 17 in K8s integration tests by default when setting spark-tgz.

Why are the changes needed?

When setting parameters spark-tgz during integration tests, the error that resource-managers/kubernetes/docker/src/main/dockerfiles/spark/Dockerfile.java17 cannot be found occurs.

This is due to the default value of spark.kubernetes.test.dockerFile is a relative path.

When using the tgz, the working directory is $UNPACKED_SPARK_TGZ, and the relative path is invalid.

Does this PR introduce any user-facing change?

No

How was this patch tested?

Runing k8s integration test manaully:

sbt

$ build/sbt -Pkubernetes -Pkubernetes-integration-tests -Dtest.exclude.tags=minikube,r "kubernetes-integration-tests/test"

KubernetesSuite:
- Run SparkPi with no resources
- Run SparkPi with no resources & statefulset allocation
- Run SparkPi with a very long application name.
- Use SparkLauncher.NO_RESOURCE
- Run SparkPi with a master URL without a scheme.
- Run SparkPi with an argument.
- Run SparkPi with custom labels, annotations, and environment variables.
- All pods have the same service account by default
- Run extraJVMOptions check on driver
- Run SparkRemoteFileTest using a remote data file
- Verify logging configuration is picked from the provided SPARK_CONF_DIR/log4j2.properties
- Run SparkPi with env and mount secrets.
- Run PySpark on simple pi.py example
- Run PySpark to test a pyfiles example
- Run PySpark with memory customization
- Run in client mode.
- Start pod creation from template
- PVs with local hostpath storage on statefulsets
- PVs with local hostpath and storageClass on statefulsets
- PVs with local storage
- Launcher client dependencies
- SPARK-33615: Launcher client archives
- SPARK-33748: Launcher python client respecting PYSPARK_PYTHON
- SPARK-33748: Launcher python client respecting spark.pyspark.python and spark.pyspark.driver.python
- Launcher python client dependencies using a zip file
- Test basic decommissioning
- Test basic decommissioning with shuffle cleanup
- Test decommissioning with dynamic allocation & shuffle cleanups
- Test decommissioning timeouts
- SPARK-37576: Rolling decommissioning
Run completed in 27 minutes, 8 seconds.
Total number of tests run: 30
Suites: completed 2, aborted 0
Tests: succeeded 30, failed 0, canceled 0, ignored 0, pending 0
All tests passed.

maven with spark-tgz

$ bash resource-managers/kubernetes/integration-tests/dev/dev-run-integration-tests.sh --spark-tgz $TARBALL_TO_TEST --exclude-tags r

KubernetesSuite:
- Run SparkPi with no resources
- Run SparkPi with no resources & statefulset allocation
- Run SparkPi with a very long application name.
- Use SparkLauncher.NO_RESOURCE
- Run SparkPi with a master URL without a scheme.
- Run SparkPi with an argument.
- Run SparkPi with custom labels, annotations, and environment variables.
- All pods have the same service account by default
- Run extraJVMOptions check on driver
- Run SparkRemoteFileTest using a remote data file
- Verify logging configuration is picked from the provided SPARK_CONF_DIR/log4j2.properties
- Run SparkPi with env and mount secrets.
- Run PySpark on simple pi.py example
- Run PySpark to test a pyfiles example
- Run PySpark with memory customization
- Run in client mode.
- Start pod creation from template
- PVs with local hostpath storage on statefulsets
- PVs with local hostpath and storageClass on statefulsets
- PVs with local storage
- Launcher client dependencies
- SPARK-33615: Launcher client archives
- SPARK-33748: Launcher python client respecting PYSPARK_PYTHON
- SPARK-33748: Launcher python client respecting spark.pyspark.python and spark.pyspark.driver.python
- Launcher python client dependencies using a zip file
- Test basic decommissioning
- Test basic decommissioning with shuffle cleanup
- Test decommissioning with dynamic allocation & shuffle cleanups
- Test decommissioning timeouts
- SPARK-37576: Rolling decommissioning
Run completed in 30 minutes, 6 seconds.
Total number of tests run: 30
Suites: completed 2, aborted 0
Tests: succeeded 30, failed 0, canceled 0, ignored 0, pending 0
All tests passed.

maven without spark-tgz

$ bash resource-managers/kubernetes/integration-tests/dev/dev-run-integration-tests.sh --exclude-tags r

KubernetesSuite:
- Run SparkPi with no resources
- Run SparkPi with no resources & statefulset allocation
- Run SparkPi with a very long application name.
- Use SparkLauncher.NO_RESOURCE
- Run SparkPi with a master URL without a scheme.
- Run SparkPi with an argument.
- Run SparkPi with custom labels, annotations, and environment variables.
- All pods have the same service account by default
- Run extraJVMOptions check on driver
- Run SparkRemoteFileTest using a remote data file
- Verify logging configuration is picked from the provided SPARK_CONF_DIR/log4j2.properties
- Run SparkPi with env and mount secrets.
- Run PySpark on simple pi.py example
- Run PySpark to test a pyfiles example
- Run PySpark with memory customization
- Run in client mode.
- Start pod creation from template
- PVs with local hostpath storage on statefulsets
- PVs with local hostpath and storageClass on statefulsets
- PVs with local storage
- Launcher client dependencies
- SPARK-33615: Launcher client archives
- SPARK-33748: Launcher python client respecting PYSPARK_PYTHON
- SPARK-33748: Launcher python client respecting spark.pyspark.python and spark.pyspark.driver.python
- Launcher python client dependencies using a zip file
- Test basic decommissioning
- Test basic decommissioning with shuffle cleanup
- Test decommissioning with dynamic allocation & shuffle cleanups
- Test decommissioning timeouts
- SPARK-37576: Rolling decommissioning
Run completed in 35 minutes, 0 seconds.
Total number of tests run: 30
Suites: completed 2, aborted 0
Tests: succeeded 30, failed 0, canceled 0, ignored 0, pending 0
All tests passed.

…etting spark-tgz

dcoliversun · 2022-02-23T10:47:06Z

cc @dongjoon-hyun @sarutak
It would be good if you could take a look when you have time, thanks!

dongjoon-hyun

Thank you for pinging me, @dcoliversun .

AmplabJenkins · 2022-02-24T09:59:19Z

Can one of the admins verify this patch?

dongjoon-hyun · 2022-02-25T01:01:55Z

resource-managers/kubernetes/integration-tests/scripts/setup-integration-test-env.sh

@@ -106,7 +106,7 @@ then
    # OpenJDK base-image tag (e.g. 8-jre-slim, 11-jre-slim)
    JAVA_IMAGE_TAG_BUILD_ARG="-b java_image_tag=$JAVA_IMAGE_TAG"
  else
-    JAVA_IMAGE_TAG_BUILD_ARG="-f $DOCKER_FILE"
+    JAVA_IMAGE_TAG_BUILD_ARG="-f $DOCKER_FILE_BASE_PATH$DOCKER_FILE"


This seems to break the build when we use a docker file as an argument. For example, what happens when we give an absolute path? Please try this patch with --docker-file option from your environment.

OK. I handle this case.

Hi. These code can check that $DOCKER_FILE is absolute path or not.

If $DOCKER_FILE is absolute path, nothing changes.

If relative path, $DOCKER_FILE_BASE_PATH and $DOCKER_FILE are joined together, for using Java 17 in K8S IT by default.

Are you saying that new commit do that, @dcoliversun ?

Yes. First commit doesn't supoort --docker-file is absolute path.

dongjoon-hyun · 2022-02-25T01:02:20Z

Also, cc @LuciferYang since this is Java 17.

LuciferYang · 2022-02-25T08:25:47Z

Is the behavior of Java 17 and Java 8 inconsistent ?

dcoliversun · 2022-02-25T14:10:49Z

Is the behavior of Java 17 and Java 8 inconsistent ?

Hi @LuciferYang . Not compatibility issue, it's about a relative path about Dockerfile.java17 is not available in spark-tgz option when integration test.

resource-managers/kubernetes/integration-tests/scripts/setup-integration-test-env.sh

LuciferYang · 2022-02-26T05:48:46Z

@dongjoon-hyun @dcoliversun Sorry for the late reply. This is the first time I run the test locally of this module, so I spent some time setup the local test environment.

I use following command to build a tarball use Zulu17.32+13-CA

dev/make-distribution.sh --tgz -Phadoop-3 -Phadoop-cloud -Pmesos -Pyarn -Pkinesis-asl -Phive-thriftserver -Pspark-ganglia-lgpl -Pkubernetes -Phive

then checkout this pr use command:

gh pr checkout 35627

run test use command as follows:

resource-managers/kubernetes/integration-tests/dev/dev-run-integration-tests.sh --spark-tgz $TARBALL_TO_TEST --exclude-tags r

when $TRBALL_TO_TEST use absolute path, this pr can fix the described issue

dcoliversun · 2022-02-26T07:03:16Z

@dongjoon-hyun @dcoliversun Sorry for the late reply. This is the first time I run the test locally of this module, so I spent some time setup the local test environment.

I use following command to build a tarball use Zulu17.32+13-CA
dev/make-distribution.sh --tgz -Phadoop-3 -Phadoop-cloud -Pmesos -Pyarn -Pkinesis-asl -Phive-thriftserver -Pspark-ganglia-lgpl -Pkubernetes -Phive
then checkout this pr use command:
gh pr checkout 35627
run test use command as follows:
resource-managers/kubernetes/integration-tests/dev/dev-run-integration-tests.sh --spark-tgz $TARBALL_TO_TEST --exclude-tags r
when $TRBALL_TO_TEST use absolute path, this pr can fix the described issue

Thanks for your verification. I'm glad this pr can fix the issue😀

dongjoon-hyun · 2022-02-26T23:08:12Z

Thank you so much again, @LuciferYang !

dongjoon-hyun

+1, LGTM. I also verified this PR manually. Thank you, @dcoliversun , @LuciferYang , and @martin-g . Merged to master for Apache Spark 3.3.0.

[SPARK-38302][K8S][TESTS] Use Java 17 in K8S integration tests when s…

3855d8d

…etting spark-tgz

github-actions bot added BUILD KUBERNETES labels Feb 23, 2022

dongjoon-hyun reviewed Feb 23, 2022

View reviewed changes

dongjoon-hyun changed the title ~~[SPARK-38302][K8S][TESTS] Use Java 17 in K8S integration tests when setting spark-tgz~~ [SPARK-38302][K8S][TESTS] Use Java 17 in K8S IT in case of spark-tgz option Feb 24, 2022

dongjoon-hyun requested changes Feb 25, 2022

View reviewed changes

allow docker-file to be absolute path

2e85e70

martin-g reviewed Feb 25, 2022

View reviewed changes

resource-managers/kubernetes/integration-tests/scripts/setup-integration-test-env.sh Show resolved Hide resolved

dongjoon-hyun approved these changes Feb 26, 2022

View reviewed changes

dongjoon-hyun closed this in 3aa0cd4 Feb 26, 2022

dcoliversun deleted the SPARK-38302 branch February 27, 2022 00:26

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-38302][K8S][TESTS] Use `Java 17` in K8S IT in case of `spark-tgz` option #35627

[SPARK-38302][K8S][TESTS] Use `Java 17` in K8S IT in case of `spark-tgz` option #35627

dcoliversun commented Feb 23, 2022 •

edited

Loading

dcoliversun commented Feb 23, 2022

dongjoon-hyun left a comment

AmplabJenkins commented Feb 24, 2022

dongjoon-hyun Feb 25, 2022

dcoliversun Feb 25, 2022

dcoliversun Feb 25, 2022

dongjoon-hyun Feb 25, 2022

dcoliversun Feb 26, 2022

dongjoon-hyun commented Feb 25, 2022

LuciferYang commented Feb 25, 2022

dcoliversun commented Feb 25, 2022

LuciferYang commented Feb 26, 2022 •

edited

Loading

dcoliversun commented Feb 26, 2022

dongjoon-hyun commented Feb 26, 2022

dongjoon-hyun left a comment

[SPARK-38302][K8S][TESTS] Use Java 17 in K8S IT in case of spark-tgz option #35627

[SPARK-38302][K8S][TESTS] Use Java 17 in K8S IT in case of spark-tgz option #35627

Conversation

dcoliversun commented Feb 23, 2022 • edited Loading

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

sbt

maven with spark-tgz

maven without spark-tgz

dcoliversun commented Feb 23, 2022

dongjoon-hyun left a comment

Choose a reason for hiding this comment

AmplabJenkins commented Feb 24, 2022

dongjoon-hyun Feb 25, 2022

Choose a reason for hiding this comment

dcoliversun Feb 25, 2022

Choose a reason for hiding this comment

dcoliversun Feb 25, 2022

Choose a reason for hiding this comment

dongjoon-hyun Feb 25, 2022

Choose a reason for hiding this comment

dcoliversun Feb 26, 2022

Choose a reason for hiding this comment

dongjoon-hyun commented Feb 25, 2022

LuciferYang commented Feb 25, 2022

dcoliversun commented Feb 25, 2022

LuciferYang commented Feb 26, 2022 • edited Loading

dcoliversun commented Feb 26, 2022

dongjoon-hyun commented Feb 26, 2022

dongjoon-hyun left a comment

Choose a reason for hiding this comment

[SPARK-38302][K8S][TESTS] Use `Java 17` in K8S IT in case of `spark-tgz` option #35627

[SPARK-38302][K8S][TESTS] Use `Java 17` in K8S IT in case of `spark-tgz` option #35627

dcoliversun commented Feb 23, 2022 •

edited

Loading

LuciferYang commented Feb 26, 2022 •

edited

Loading