[VL][CI] Change to use JDK-17 for Spark 3.3/3.4/3.5 tests#9209
[VL][CI] Change to use JDK-17 for Spark 3.3/3.4/3.5 tests#9209PHILO-HE merged 7 commits intoapache:mainfrom
Conversation
|
Thanks for opening a pull request! Could you open an issue for this pull request on Github Issues? https://github.com/apache/incubator-gluten/issues Then could you also rename commit message and pull request title in the following format? See also: |
|
We need to document the JDK support policy in the index page. |
| $MVN_CMD clean test -Pspark-3.5 -Pbackends-velox -Piceberg -Pdelta -Phudi -Pspark-ut \ | ||
| -DargLine="-Dspark.test.home=/opt/shims/spark35/spark_home/" \ | ||
| -DtagsToExclude=org.apache.spark.tags.ExtendedSQLTest,org.apache.gluten.tags.UDFTest,org.apache.gluten.tags.SkipTestTags | ||
| -DargLine="-Dspark.test.home=$SPARK_HOME ${EXTRA_FLAGS}" |
| spark-test-spark35: | ||
| needs: build-native-lib-centos-7 | ||
| runs-on: ubuntu-20.04 | ||
| container: apache/gluten:centos-8-jdk8 |
There was a problem hiding this comment.
should change with jdk17 image for this job also
There was a problem hiding this comment.
JDK-17 is installed and set to use for each JDK-17 jobs.
I note arrow jars generated by build-native-lib job are actually compiled by JDK-8 and they are reused by other JDK-17 jobs. It works well. So currently we don't need to use JDK-17 docker inside which arrow jars are compiled by JDK-17.
Maybe, we can just keep one docker where both JDK-11 & 17 installed and switch the JDK to the required one for different CI jobs. For now, we can keep the two dockers and observe the CI for some time.
There was a problem hiding this comment.
thanks, I see the point now, it seems for now we need to use a combination of JDK8(arrow jar) + JDK17 spark jars which are little odd. will try to check this on #9218
|
Run Gluten Clickhouse CI on x86 |
|
Run Gluten Clickhouse CI on x86 |
1 similar comment
|
Run Gluten Clickhouse CI on x86 |
|
Run Gluten Clickhouse CI on x86 |
* [GLUTEN-9199][VL] Fix error when creating shuffle file: open with O_CREAT or O_TMPFILE in second argument needs 3 arguments (#9200) * [DOC] Fix broken links in documents (#9320) * [GLUTEN-9369][DOC] Fix commands in the Gluten C++ debugging developer doc (#9368) * [VL][CI] Change to use JDK-17 for Spark 3.3/3.4/3.5 tests (#9209) * [VL][CI] Bump to use ubuntu-22.04 runner (#9262) --------- Co-authored-by: Hongze Zhang <hongze.zzz123@gmail.com> Co-authored-by: PHILO-HE <philo@apache.org> Co-authored-by: Dina Suehiro Jones <dina.s.jones@intel.com> Co-authored-by: Yuan <yuanzhou@apache.org>
2 participants
What changes were proposed in this pull request?
According to CI usage feedback, Spark 3.3/3.4/3.5 tests requires around 3 hours (see #9188). By upgrading to JDK-17, the test time can be reduced a lot.
We will continue the support for JDK-8. In CI, all Spark-3.2 UTs and some TPC tests are still verified on JDK-8, which should be sufficient to uncover code issues on JDK-8, if any.
(Fixes: #9188)