Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix reading parquet column with unused dictionary #15942

Merged
merged 3 commits into from
Feb 2, 2023
Merged

Fix reading parquet column with unused dictionary #15942

merged 3 commits into from
Feb 2, 2023

Commits on Feb 2, 2023

  1. Remove unnecessary ColumnChunkDescriptor class

    @raunaqmorarka
    raunaqmorarka committed Feb 2, 2023
    Configuration menu
    Copy the full SHA
    b69c787 View commit details
    Browse the repository at this point in the history

  2. Convert DataPage into a sealed class

    @raunaqmorarka
    raunaqmorarka committed Feb 2, 2023
    Configuration menu
    Copy the full SHA
    a68b2b4 View commit details
    Browse the repository at this point in the history

  3. Fix reading parquet column with unused dictionary 

    A parquet file produced by Impala was found to have an empty dictionary
which is not used in the encoding of data pages in the column.
For such a case we cannot rely on ColumnChunkMetaData#hasDictionaryPage
as that checks for whether the data pages are also encoded using the dictionary.
This change removes usage of hasDictionaryPage to fix query failures
with such files.
    @raunaqmorarka
    raunaqmorarka committed Feb 2, 2023
    Configuration menu
    Copy the full SHA
    31b14d9 View commit details
    Browse the repository at this point in the history