-
Notifications
You must be signed in to change notification settings - Fork 418
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[GLUTEN-3559][VL] Rewrite GlutenInsertSuite test cases with default values #4737
Conversation
Run Gluten Clickhouse CI |
@Surbhi-Vijay An issue for this can be opened: Velox does not support back filling the existing records while scan |
@Surbhi-Vijay, could you please rebase the code and resolve the conflicts? Thanks! |
a4ebd8d
to
0333858
Compare
Run Gluten Clickhouse CI |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for your efforts!
@PHILO-HE All checks have passed. |
===== Performance report for TPCH SF2000 with Velox backend, for reference only ====
|
===== Performance report for TPCH SF2000 with Velox backend, for reference only ====
|
What changes were proposed in this pull request?
Additional support was added in Spark-3.4 for default values in parquet file scans.
https://issues.apache.org/jira/browse/SPARK-39265
While scanning the files if the column with default value does not have any value then reader appends the default value to it. So, even if the column with default value was added later, file scan still provides values for all records (existing as well as new ones).
Velox does not support back filling the existing records while scan. So, if the column with default value was added later then it will provide null as column value for existing records.
This is a behavior difference and not an inconsistent behavior. Users can update the existing data by running DML commands.
This PR, rewrites those testcases with default value in Gluten.
(Fixes: #3559)
How was this patch tested?
Unit tests are passing