Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Parquet: Support reading INT96 column in row group filter #8988

Merged
merged 1 commit into from Jan 8, 2024
Merged

Parquet: Support reading INT96 column in row group filter #8988

merged 1 commit into from Jan 8, 2024

Conversation

manuzhang
Copy link
Contributor

@manuzhang manuzhang commented Nov 5, 2023 •
edited

This fixes #8990

@manuzhang
Copy link
Contributor Author

@yabola @nastra PTAL, thanks!

@manuzhang manuzhang changed the title Spark 3.5: Don't throw exception when decoding dictionary of type INT96 Parquet: don't throw exception on row group filters when reading INT96 column Nov 6, 2023
@manuzhang manuzhang changed the title Parquet: don't throw exception on row group filters when reading INT96 column Parquet: Support reading INT96 column in row group filter Nov 7, 2023
nastra
nastra approved these changes Nov 7, 2023
View reviewed changes
Copy link
Contributor

@nastra nastra left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM with one suggestion and once CI passes

@nastra
Copy link
Contributor

nastra commented Nov 7, 2023

@manuzhang does that also fix #8949 or is that a different root cause?

@manuzhang
Copy link
Contributor Author

@nastra #8949 is a different RC.

@manuzhang
Copy link
Contributor Author

Any more comments?

@nastra
Copy link
Contributor

nastra commented Nov 9, 2023

I think it would be good to also get the opinion of @RussellSpitzer or @aokolnychyi on this one

@manuzhang
Copy link
Contributor Author

@RussellSpitzer @aokolnychyi could you please take a look?

@manuzhang
Copy link
Contributor Author

gentle ping @nastra @RussellSpitzer @aokolnychyi

@manuzhang
Copy link
Contributor Author

I submitted #9408 to fix failed flaky test.

@manuzhang
Copy link
Contributor Author

@nastra can we move forward with this PR?

@manuzhang manuzhang force-pushed the fix_decode_int96 branch 2 times, most recently from dce3a37 to 12c2e6e Compare January 6, 2024 14:30
@manuzhang manuzhang requested a review from nastra January 6, 2024 15:43
@manuzhang manuzhang force-pushed the fix_decode_int96 branch 3 times, most recently from 2474e37 to cc8b63b Compare January 7, 2024 15:28
@nastra nastra merged commit 4a99678 into apache:main Jan 8, 2024
41 checks passed
@manuzhang manuzhang deleted the fix_decode_int96 branch January 8, 2024 11:08
geruh pushed a commit to geruh/iceberg that referenced this pull request Jan 26, 2024
@manuzhang @geruh
Parquet: Support reading INT96 column in row group filter (apache#8988)
6118f2d
adnanhemani pushed a commit to adnanhemani/iceberg that referenced this pull request Jan 30, 2024
@manuzhang @adnanhemani
Parquet: Support reading INT96 column in row group filter (apache#8988)
35d2884
devangjhabakh pushed a commit to cdouglas/iceberg that referenced this pull request Apr 22, 2024
@manuzhang @devangjhabakh
Parquet: Support reading INT96 column in row group filter (apache#8988)
52bbb27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@nastra nastra nastra approved these changes

@RussellSpitzer RussellSpitzer Awaiting requested review from RussellSpitzer

@aokolnychyi aokolnychyi Awaiting requested review from aokolnychyi

Assignees
No one assigned
Labels
parquet spark
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Cannot decode dictionary of type INT96 when reading imported Spark parquet table
2 participants
@manuzhang @nastra