[GLUTEN-3359] Add Spark3.4 unit test framework #3497
Conversation
|
Run Gluten Clickhouse CI |
|
Run Gluten Clickhouse CI |
|
Run Gluten Clickhouse CI |
|
Run Gluten Clickhouse CI |
2 similar comments
|
Run Gluten Clickhouse CI |
|
Run Gluten Clickhouse CI |
|
Run Gluten Clickhouse CI |
4 similar comments
|
Run Gluten Clickhouse CI |
|
Run Gluten Clickhouse CI |
|
Run Gluten Clickhouse CI |
|
Run Gluten Clickhouse CI |
|
Run Gluten Clickhouse CI |
2 similar comments
|
Run Gluten Clickhouse CI |
|
Run Gluten Clickhouse CI |
JkSelf
changed the title
|
Run Gluten Clickhouse CI |
|
Run Gluten Clickhouse CI |
|
Run Gluten Clickhouse CI |
|
Run Gluten Clickhouse CI |
| - name: Build and Run unit test for Spark 3.4.1(slow tests) | ||
| run: | | ||
| docker exec ubuntu2004-test-spark34-slow-$GITHUB_RUN_ID bash -l -c 'cd /opt/gluten && \ | ||
| mvn clean install -Pspark-3.4 -Pbackends-velox -Prss -Pspark-ut -DargLine="-Dspark.test.home=/opt/spark331" -DtagsToInclude=org.apache.spark.tags.ExtendedSQLTest' |
There was a problem hiding this comment.
looks like we should target for spark341 here?
There was a problem hiding this comment.
@zhouyuan Yes. Only GlutenParquetIOSuite#SPARK-40128 read DELTA_LENGTH_BYTE_ARRAY encoded strings test throws a Path does not exist exception when reading the input parquet data. However, even when using Spark 3.4.1, it still fails with a velox exception. Therefore, can we use Spark 3.3.1 in this PR and use Spark 3.4.1 in the subsequent PR?
JkSelf
changed the title
|
===== Performance report for TPCH SF2000 with Velox backend, for reference only ====
|
What changes were proposed in this pull request?
This PR consists of two commits. The first commit involves copying the spark 33 unit test into spark 34 within gluten-ut. The second commit addresses the compilation issues and disables the failed unit test. And the disabled unit test is recorded here.
How was this patch tested?
Adding spark34 folder in gluten-ut module.