Task Description
What needs to be done:
Reduce unnecessary row group metadata loading
Why this task is needed:
For a footer of a parquet file, it is mainly divided into the metadata of the file and the metadata of the row group. Generally, the proportion of the metadata of the row group is much larger than that of the file. Moreover, in some wide table practices, the metadata of the row group increases linearly as the number of columns increases. Therefore, for some places in the code logic where metadata needs to be loaded, it is necessary to reconsider whether the metadata of the row group also needs to be loaded. Currently, the logic directly loads the entire footer, which will cause unnecessary IO and deserialization for some scenarios that only require file metadata.
Task Type
Performance optimization
Related Issues
Parent feature issue: (if applicable )
Related issues:
NOTE: Use Relationships button to add parent/blocking issues after issue is created.
Task Description
What needs to be done:
Reduce unnecessary row group metadata loading
Why this task is needed:
For a footer of a parquet file, it is mainly divided into the metadata of the file and the metadata of the row group. Generally, the proportion of the metadata of the row group is much larger than that of the file. Moreover, in some wide table practices, the metadata of the row group increases linearly as the number of columns increases. Therefore, for some places in the code logic where metadata needs to be loaded, it is necessary to reconsider whether the metadata of the row group also needs to be loaded. Currently, the logic directly loads the entire footer, which will cause unnecessary IO and deserialization for some scenarios that only require file metadata.
Task Type
Performance optimization
Related Issues
Parent feature issue: (if applicable )
Related issues:
NOTE: Use
Relationshipsbutton to add parent/blocking issues after issue is created.