Skip to content
This repository has been archived by the owner on Feb 18, 2024. It is now read-only.

Added support for projection pushdown on IPC files #264

Merged
merged 5 commits into from Aug 11, 2021
Merged

Conversation

jorgecarleitao
Copy link
Owner

@jorgecarleitao jorgecarleitao commented Aug 9, 2021

Closes #261 and #264 and simplifies internal code (to support this).

Backwards incompatible changes

  • ipc::read::FileReader::new now accept a new parameter projection: Option<Vec<usize>> that can be used to filter columns.
  • read::ipc::read_record_batch now accept a new parameter projection: Option<(&[usize], Arc<Schema>)> that can be used to filter columns.

where usize is the index of the column in the IPC schema (known from the file metadata).

Columns not selected are neither read nor deserialized (i.e. are skipped). Pass None to recover the original behavior.

@codecov
Copy link

codecov bot commented Aug 9, 2021

Codecov Report

Merging #264 (80ec1e3) into main (d545253) will increase coverage by 0.46%.
The diff coverage is 92.19%.

Impacted file tree graph

@@            Coverage Diff             @@
##             main     #264      +/-   ##
==========================================
+ Coverage   76.76%   77.22%   +0.46%     
==========================================
  Files         232      252      +20     
  Lines       19820    21792    +1972     
==========================================
+ Hits        15214    16829    +1615     
- Misses       4606     4963     +357     
Impacted Files Coverage Δ
arrow-flight/src/utils.rs 0.00% <ø> (ø)
...ng/src/flight_server_scenarios/integration_test.rs 0.00% <ø> (ø)
src/io/ipc/read/array/null.rs 50.00% <50.00%> (ø)
src/io/ipc/read/array/fixed_size_list.rs 58.33% <58.33%> (ø)
src/io/ipc/read/deserialize.rs 94.31% <80.76%> (-1.43%) ⬇️
src/io/ipc/read/read_basic.rs 88.37% <88.37%> (ø)
src/io/ipc/read/reader.rs 87.58% <89.47%> (+0.91%) ⬆️
src/io/ipc/read/array/binary.rs 100.00% <100.00%> (ø)
src/io/ipc/read/array/boolean.rs 100.00% <100.00%> (ø)
src/io/ipc/read/array/dictionary.rs 100.00% <100.00%> (ø)
... and 52 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d545253...80ec1e3. Read the comment docs.

src/io/ipc/read/reader.rs Outdated Show resolved Hide resolved
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
feature A new feature
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Allow projection pushdown to IPC files
3 participants