Skip to content
This repository has been archived by the owner on Feb 18, 2024. It is now read-only.

Added support to read and write extension types to and from parquet #396

Merged
merged 2 commits into from
Sep 12, 2021

Conversation

jorgecarleitao
Copy link
Owner

@jorgecarleitao jorgecarleitao commented Sep 11, 2021

Extensions can now be written and read to and from parquet (roundtrip), thereby allowing the extension to persist, just like IPC. This is achieved by using the extension available in the schema to derive the arrays' extension.

Closes #392

@jorgecarleitao jorgecarleitao added the feature A new feature label Sep 11, 2021
@codecov
Copy link

codecov bot commented Sep 11, 2021

Codecov Report

Merging #396 (357ad2b) into main (f326fed) will increase coverage by 0.02%.
The diff coverage is 100.00%.

Impacted file tree graph

@@            Coverage Diff             @@
##             main     #396      +/-   ##
==========================================
+ Coverage   81.13%   81.15%   +0.02%     
==========================================
  Files         347      347              
  Lines       21991    22003      +12     
==========================================
+ Hits        17842    17857      +15     
+ Misses       4149     4146       -3     
Impacted Files Coverage Δ
src/datatypes/mod.rs 71.42% <100.00%> (+1.93%) ⬆️
src/io/ipc/convert.rs 94.95% <100.00%> (+0.04%) ⬆️
src/io/parquet/read/binary/dictionary.rs 81.66% <100.00%> (+0.63%) ⬆️
src/io/parquet/read/fixed_size_binary.rs 51.28% <100.00%> (+0.64%) ⬆️
src/io/parquet/read/mod.rs 53.01% <100.00%> (+1.79%) ⬆️
src/io/parquet/write/dictionary.rs 72.72% <100.00%> (ø)
src/io/parquet/write/mod.rs 51.28% <100.00%> (ø)
src/io/parquet/write/schema.rs 54.58% <100.00%> (ø)
tests/it/io/ipc/write/file.rs 100.00% <100.00%> (ø)
tests/it/io/parquet/mod.rs 94.90% <100.00%> (+0.02%) ⬆️
... and 1 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f326fed...357ad2b. Read the comment docs.

@jorgecarleitao jorgecarleitao merged commit 62ba59e into main Sep 12, 2021
@jorgecarleitao jorgecarleitao deleted the extension_parquet branch September 12, 2021 19:26
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
feature A new feature
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Support read/write extension datatype in parquet io
1 participant