Skip to content

[GLUTEN-11550][UT] Fix 2 TPCDS traits, enable 8 disabled test suites for Spark 4.x#11816

Merged
zhouyuan merged 7 commits intoapache:mainfrom
baibaichen:fix/11550-enable-all
Apr 10, 2026
Merged

[GLUTEN-11550][UT] Fix 2 TPCDS traits, enable 8 disabled test suites for Spark 4.x#11816
zhouyuan merged 7 commits intoapache:mainfrom
baibaichen:fix/11550-enable-all

Conversation

@baibaichen
Copy link
Copy Markdown
Contributor

@baibaichen baibaichen commented Mar 24, 2026

What changes were proposed in this pull request?

Fix 2 TPCDS suite traits, disable 1 wrong-trait suite, and enable 8 previously disabled Spark 4.x test suites by adding Gluten-aware testGluten overrides and fixing test configurations.

Trait fixes + disable (commit 1):

Suite Fix
GlutenTPCDSQueryWithStatsSuite trait fix: GlutenTestsCommonTrait -> GlutenSQLTestsTrait
GlutenTPCDSQueryANSISuite trait fix: GlutenTestsCommonTrait -> GlutenSQLTestsTrait
GlutenStreamingQueryHashPartitionVerifySuite disable (wrong trait, runs as vanilla Spark)

Suites enabled (8):

Suite Fix type
GlutenCsvExpressionsSuite glutenCheckExpression unwrap SparkException
GlutenWholeTextFileV1Suite testFile() override with getWorkspaceFilePath
GlutenWholeTextFileV2Suite testFile() override with getWorkspaceFilePath
GlutenSparkPlanSuite testGluten (ColumnarToRowExecBase)
GlutenInsertSortForLimitAndOffsetSuite testGluten (SortExecTransformer, TakeOrderedAndProjectExecTransformer)
GlutenJoinHintSuite testGluten (CartesianProductExecTransformer)
GlutenDataSourceScanExecRedactionSuite testGluten (FileSourceScanExecTransformer)
GlutenDataSourceV2ScanExecRedactionSuite testGluten (BatchScanExecTransformer)

Root cause categories:

  1. Operator replacement (5 suites): Gluten replaces Spark operators with Transformer variants — testGluten overrides match the correct class names
  2. Path/resource (2 suites): WholeTextFile suites need getWorkspaceFilePath override
  3. Exception wrapping (1 suite): CsvExpressions needs SparkException unwrapping

Every .exclude() has a matching testGluten override (11 total).

How was this patch tested?

All suites verified locally with SPARK_ANSI_SQL_MODE=false using:

./dev/run-scala-test.sh --mvnd -Pjava-17,spark-4.1,scala-2.13,backends-velox,hadoop-3.3,spark-ut,delta -pl gluten-ut/spark41 -s <SuiteName>

Related issue: #11550

@github-actions github-actions bot added the CORE works for Gluten Core label Mar 24, 2026
@baibaichen baibaichen changed the title [GLUTEN-11550][UT] Enable 13 disabled test suites for Spark 4.0/4.1 [GLUTEN-11550][UT] Enable 30 disabled test suites for Spark 4.0/4.1 Mar 24, 2026
@github-actions
Copy link
Copy Markdown

Run Gluten Clickhouse CI on x86

@baibaichen baibaichen force-pushed the fix/11550-enable-all branch from 5782b90 to 451760d Compare March 26, 2026 07:30
@github-actions
Copy link
Copy Markdown

Run Gluten Clickhouse CI on x86

1 similar comment
@github-actions
Copy link
Copy Markdown

Run Gluten Clickhouse CI on x86

@baibaichen baibaichen force-pushed the fix/11550-enable-all branch from 431721f to bb137ad Compare April 10, 2026 04:50
@baibaichen baibaichen changed the title [GLUTEN-11550][UT] Enable 30 disabled test suites for Spark 4.0/4.1 [GLUTEN-11550][UT] Enable 18 disabled test suites for Spark 4.x Apr 10, 2026
@github-actions
Copy link
Copy Markdown

Run Gluten Clickhouse CI on x86

@github-actions
Copy link
Copy Markdown

Run Gluten Clickhouse CI on x86

@baibaichen baibaichen force-pushed the fix/11550-enable-all branch 2 times, most recently from 8ec0857 to 0c2ca3d Compare April 10, 2026 05:10
@github-actions
Copy link
Copy Markdown

Run Gluten Clickhouse CI on x86

1 similar comment
@github-actions
Copy link
Copy Markdown

Run Gluten Clickhouse CI on x86

@baibaichen baibaichen force-pushed the fix/11550-enable-all branch from 0c2ca3d to d9b6122 Compare April 10, 2026 05:22
@github-actions
Copy link
Copy Markdown

Run Gluten Clickhouse CI on x86

@baibaichen baibaichen force-pushed the fix/11550-enable-all branch from d9b6122 to 666ce08 Compare April 10, 2026 05:52
@github-actions
Copy link
Copy Markdown

Run Gluten Clickhouse CI on x86

@baibaichen baibaichen changed the title [GLUTEN-11550][UT] Enable 18 disabled test suites for Spark 4.x [GLUTEN-11550][UT] Enable 15 disabled test suites for Spark 4.x Apr 10, 2026
@baibaichen baibaichen force-pushed the fix/11550-enable-all branch from 666ce08 to 9b8fd96 Compare April 10, 2026 10:10
@baibaichen baibaichen changed the title [GLUTEN-11550][UT] Enable 15 disabled test suites for Spark 4.x [GLUTEN-11550][UT] Enable 10 disabled test suites for Spark 4.x (with testGluten rewrites) Apr 10, 2026
@github-actions
Copy link
Copy Markdown

Run Gluten Clickhouse CI on x86

@baibaichen baibaichen force-pushed the fix/11550-enable-all branch from 9b8fd96 to 169b334 Compare April 10, 2026 12:27
@github-actions
Copy link
Copy Markdown

Run Gluten Clickhouse CI on x86

baibaichen and others added 3 commits April 10, 2026 12:47
…nges)

Fix trait GlutenTestsCommonTrait -> GlutenSQLTestsTrait:
- GlutenTPCDSQueryWithStatsSuite
- GlutenTPCDSQueryANSISuite

Disable GlutenStreamingQueryHashPartitionVerifySuite (wrong trait, runs as vanilla Spark)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Wrap exception in glutenCheckExpression with fail() to match Spark's
checkEvaluationWithoutCodegen behavior. No testGluten override needed.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…extFileV2Suite

Override testFile() to use getWorkspaceFilePath() instead of default
jar-based path. The default testFile() returns jar:file: URI which
Hadoop Path does not support. Same pattern used by GlutenCSVSuite,
GlutenJsonSuite, GlutenParquetIOSuite.
V1 3/3 passed, V2 3/3 passed.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
baibaichen and others added 4 commits April 10, 2026 12:47
Override test to find ColumnarToRowExecBase instead of ColumnarToRowExec.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
6 tests rewritten with testGluten (match TakeOrderedAndProjectExecTransformer,
LimitExecTransformer, ColumnarCollectLimitBaseExec). Original tests excluded.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
testGluten for shuffle-replicate-nl matching CartesianProductExecTransformer.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
… testGluten

Enable and fix:
- GlutenDataSourceScanExecRedactionSuite: testGluten matching FileSourceScanExecTransformer
- GlutenDataSourceV2ScanExecRedactionSuite: testGluten matching BatchScanExecTransformer
- VTS: enable JoinHint with exclude, enable DataSource suites with excludes

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@baibaichen baibaichen force-pushed the fix/11550-enable-all branch from 169b334 to 1c96a55 Compare April 10, 2026 12:51
@baibaichen baibaichen changed the title [GLUTEN-11550][UT] Enable 10 disabled test suites for Spark 4.x (with testGluten rewrites) [GLUTEN-11550][UT] Fix 2 TPCDS traits, enable 8 disabled test suites for Spark 4.x Apr 10, 2026
@github-actions
Copy link
Copy Markdown

Run Gluten Clickhouse CI on x86

@zhouyuan zhouyuan merged commit c15283b into apache:main Apr 10, 2026
107 of 108 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CORE works for Gluten Core

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants