Skip to content
This repository has been archived by the owner on Feb 18, 2024. It is now read-only.

Improved parquet read performance #1124

Merged
merged 1 commit into from
Jun 29, 2022
Merged

Improved parquet read performance #1124

merged 1 commit into from
Jun 29, 2022

Conversation

jorgecarleitao
Copy link
Owner

See performance details here and tip by Kornel at https://users.rust-lang.org/t/how-to-efficiently-re-use-a-vec-u8-for-multiple-read-read/77628/6

Essentially, we no longer zero the allocation when reading.

@jorgecarleitao jorgecarleitao added the enhancement An improvement to an existing feature label Jun 29, 2022
@codecov
Copy link

codecov bot commented Jun 29, 2022

Codecov Report

Merging #1124 (e89fe34) into main (f57dbd5) will decrease coverage by 0.00%.
The diff coverage is 78.57%.

@@            Coverage Diff             @@
##             main    #1124      +/-   ##
==========================================
- Coverage   83.38%   83.38%   -0.01%     
==========================================
  Files         367      367              
  Lines       35566    35574       +8     
==========================================
+ Hits        29658    29663       +5     
- Misses       5908     5911       +3     
Impacted Files Coverage Δ
src/error.rs 42.50% <0.00%> (-3.45%) ⬇️
src/io/parquet/read/row_group.rs 99.42% <100.00%> (+0.01%) ⬆️
src/array/binary/mod.rs 88.88% <0.00%> (-1.24%) ⬇️
src/io/ipc/read/reader.rs 96.02% <0.00%> (-0.37%) ⬇️
src/array/utf8/mod.rs 86.58% <0.00%> (+0.95%) ⬆️
src/bitmap/utils/slice_iterator.rs 98.78% <0.00%> (+1.21%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f57dbd5...e89fe34. Read the comment docs.

@jorgecarleitao jorgecarleitao merged commit 81ab424 into main Jun 29, 2022
@jorgecarleitao jorgecarleitao deleted the improve_read branch June 29, 2022 15:57
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
enhancement An improvement to an existing feature
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant