Skip to content
This repository has been archived by the owner on Feb 18, 2024. It is now read-only.

Added support to filter parquet pages. #256

Merged
merged 1 commit into from
Aug 11, 2021
Merged

Added support to filter parquet pages. #256

merged 1 commit into from
Aug 11, 2021

Conversation

jorgecarleitao
Copy link
Owner

@jorgecarleitao jorgecarleitao commented Aug 5, 2021

This is the corresponding of jorgecarleitao/parquet2#34, adding support to skip parquet pages based on its statistics.

The page filter's signature is Arc<dyn Fn(&ColumnDescriptor, &DataPageHeader) -> bool>: use the first argument to tell which column is being considered and the second argument for information about the page.

In particular, DataPageHeader::statistics() contains the page statistics.

Backwards incompatible changes:

See #270

@jorgecarleitao jorgecarleitao changed the title Added support for filter of parquet pages. Added support to filter parquet pages. Aug 5, 2021
@codecov
Copy link

codecov bot commented Aug 5, 2021

Codecov Report

Merging #256 (779b7bd) into main (bfb4910) will increase coverage by 0.00%.
The diff coverage is 93.75%.

Impacted file tree graph

@@           Coverage Diff           @@
##             main     #256   +/-   ##
=======================================
  Coverage   77.37%   77.37%           
=======================================
  Files         232      232           
  Lines       20108    20119   +11     
=======================================
+ Hits        15558    15568   +10     
- Misses       4550     4551    +1     
Impacted Files Coverage Δ
src/io/parquet/read/record_batch.rs 88.15% <87.50%> (-0.25%) ⬇️
src/io/parquet/mod.rs 94.06% <100.00%> (+0.11%) ⬆️
src/io/parquet/read/mod.rs 86.08% <100.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update bfb4910...779b7bd. Read the comment docs.

Cargo.toml Outdated Show resolved Hide resolved
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
feature A new feature
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant