Skip to content
This repository has been archived by the owner on Feb 18, 2024. It is now read-only.

Added support to parquet IO on wasm32 #239

Merged
merged 1 commit into from
Aug 3, 2021
Merged

Added support to parquet IO on wasm32 #239

merged 1 commit into from
Aug 3, 2021

Conversation

jorgecarleitao
Copy link
Owner

@jorgecarleitao jorgecarleitao commented Jul 30, 2021

This PR adds a new cargo feature, io_parquet_compression, that activates the compressions in parquet.

The corollary is that this allows compiling this crate against wasm32 with parquet io. AFAIK this effectively makes arrow2 to be usable to interact with parquet directly on a browser.

As an example, this seems to allow building a client-side web application that reads or writes (uncompressed) parquet files.

For compression support, some work is needed on the zstd.

@jorgecarleitao jorgecarleitao added the enhancement An improvement to an existing feature label Jul 30, 2021
@jorgecarleitao
Copy link
Owner Author

cc @domoritz since if I remember, you were interested in the wasm aspect.

@codecov
Copy link

codecov bot commented Jul 30, 2021

Codecov Report

Merging #239 (f7194d3) into main (3cd8cff) will increase coverage by 0.16%.
The diff coverage is n/a.

Impacted file tree graph

@@            Coverage Diff             @@
##             main     #239      +/-   ##
==========================================
+ Coverage   76.82%   76.98%   +0.16%     
==========================================
  Files         229      229              
  Lines       19586    20058     +472     
==========================================
+ Hits        15046    15441     +395     
- Misses       4540     4617      +77     
Impacted Files Coverage Δ
src/compute/filter.rs 55.93% <0.00%> (-7.26%) ⬇️
src/bitmap/immutable.rs 85.56% <0.00%> (-1.28%) ⬇️
src/array/growable/primitive.rs 100.00% <0.00%> (ø)
src/bitmap/utils/slice_iterator.rs 96.42% <0.00%> (+0.15%) ⬆️
src/array/growable/list.rs 99.40% <0.00%> (+1.00%) ⬆️
src/array/growable/binary.rs 99.08% <0.00%> (+1.71%) ⬆️
src/array/growable/mod.rs 36.45% <0.00%> (+1.76%) ⬆️
src/array/growable/boolean.rs 87.71% <0.00%> (+2.00%) ⬆️
src/compute/merge_sort/mod.rs 96.09% <0.00%> (+2.53%) ⬆️
src/array/growable/fixed_binary.rs 89.34% <0.00%> (+5.59%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3cd8cff...f7194d3. Read the comment docs.

@domoritz
Copy link
Contributor

Yes, this change is awesome.

I have to rewrite my arrow-wasm repo against arrow2 but ran into some issue where some types were not exported. I'll try again sometime and consolidate the things I find in an issue.

@jorgecarleitao jorgecarleitao merged commit fa3c2ce into main Aug 3, 2021
@jorgecarleitao jorgecarleitao deleted the parquet_wasm branch August 3, 2021 04:55
@jorgecarleitao jorgecarleitao added feature A new feature and removed enhancement An improvement to an existing feature labels Aug 11, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
feature A new feature
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants