[GLUTEN-11683][VL] Fix SPARK-18108 and parquet-thrift compatibility#11689
Draft
baibaichen wants to merge 4 commits intoapache:mainfrom
Draft
[GLUTEN-11683][VL] Fix SPARK-18108 and parquet-thrift compatibility#11689baibaichen wants to merge 4 commits intoapache:mainfrom
baibaichen wants to merge 4 commits intoapache:mainfrom
Conversation
|
Run Gluten Clickhouse CI on x86 |
baibaichen
force-pushed
the
pr2/fix-parquet-thrift-spark18108
branch
from
0c5caa5 to
0728011
Compare
|
Run Gluten Clickhouse CI on x86 |
baibaichen
force-pushed
the
pr2/fix-parquet-thrift-spark18108
branch
from
0728011 to
33e87af
Compare
|
Run Gluten Clickhouse CI on x86 |
baibaichen
force-pushed
the
pr2/fix-parquet-thrift-spark18108
branch
from
33e87af to
4fede06
Compare
Replace OAP commit [15173][15343] (INT narrowing) with upstream Velox PR #15173 (fix reading array of row) to fix parquet-thrift compatibility. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…olumns When Gluten creates HiveTableHandle, it was passing all columns (including partition columns) as dataColumns. This caused Velox's convertType() to validate partition column types against the Parquet file's physical types, failing when they differ (e.g., LongType in file vs IntegerType from partition inference). Fix: build dataColumns excluding partition columns (ColumnType::kPartitionKey). Partition column values come from the partition path, not from the file. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
With OAP INT narrowing commit replaced by upstream Velox PR #15173: - Remove 2 excludes now passing: LongType->IntegerType, LongType->DateType - Add 2 excludes for new failures: IntegerType->ShortType (OAP removed) Exclude 63 (net unchanged: -2 +2). Test results: 21 pass / 63 ignored. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
baibaichen
force-pushed
the
pr2/fix-parquet-thrift-spark18108
branch
from
8c11878 to
2ec5651
Compare
|
Run Gluten Clickhouse CI on x86 |
These tests regress after skipping OAP commit 8c2bd0849 (Allow reading integers into smaller-range types). They will be re-enabled in PR3 when Velox widening commits are applied. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
baibaichen
force-pushed
the
pr2/fix-parquet-thrift-spark18108
branch
from
2ec5651 to
4bef8b1
Compare
|
Run Gluten Clickhouse CI on x86 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Fix SPARK-18108 and parquet-thrift compatibility issues.
Changes
Velox: Replace OAP INT narrowing with upstream PR #15173 (
get-velox.sh):Fix parquet-thrift compatibility by importing upstream fix.
Fix SPARK-18108 (
SubstraitToVeloxPlan.cc):Exclude partition columns from
HiveTableHandle.dataColumns()to prevent type validation failures when partition column types differ from file column types.Update VeloxTestSettings (
spark40 + spark41):Test Results
Additionally fixed (not in TypeWideningSuite):
Depends on #11684 (PR1).
Fixes #11683
How was this patch tested?
Local tests: TypeWideningSuite 21 pass / 63 ignored, SPARK-18108 ✅, parquet-thrift ✅.
Was this patch authored or co-authored using generative AI tooling?
Yes, co-authored with GitHub Copilot.