Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support building object_store and parquet on wasm32-unknown-unknown target #2896

Merged
merged 2 commits into from Oct 26, 2022

Conversation

jondo2010
Copy link
Contributor

@jondo2010 jondo2010 commented Oct 19, 2022

Which issue does this PR close?

Closes #180.

Rationale for this change

My goal is to get datafusion building and running on the wasm32-unknown-unknown target, and getting object_store and parquet are the first steps.

What changes are included in this PR?

This PR adds some conditional compilation in object_store (mostly around disabling local filesystem access), as well as shifting some dependencies to dev, and removing un-used features in deps (i.e. tokio).

Are there any user-facing changes?

None expected, this is purely a build-support change.

@github-actions github-actions bot added object-store Object Store Interface parquet Changes to the parquet crate labels Oct 19, 2022
@jondo2010 jondo2010 changed the title Support building object_store on wasm32-unknown-unknown target Support building object_store and parquet on wasm32-unknown-unknown target Oct 24, 2022
@jondo2010 jondo2010 marked this pull request as ready for review October 24, 2022 09:04
@@ -119,6 +119,9 @@ jobs:
- name: Check compilation --all-targets --no-default-features --features json
run: |
cargo check -p parquet --all-targets --no-default-features --features json
- name: Check compilation wasm32-unknown-unknown
run: |
cargo check -p parquet --no-default-features --target wasm32-unknown-unknown
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FYI compiling with lz4 appears to fail

@tustvold
Copy link
Contributor

Trying to compile object_store with non-default features returns an error about getrandom which appears related to https://docs.rs/getrandom/latest/getrandom/#webassembly-support

I'm not familiar enough with WASM to know if this is an issue or not?

* Added cargo check step to parquet workflow for wasm32-unknown-unknown
* Added compile-time warning for unsupported cloud features when compiling with wasm32
* Added cargo check features to the parquet github workflow.
* Added a section to the README.md for parquet
@jondo2010
Copy link
Contributor Author

Yep, wasm32-unknown-unknown is only going to support a limited set of features on parquet and object_store, in particular, the cloud storage features in object_store won't work, and some of the compression features on parquet won't work. I've documented everything in the respective READMEs now, and I also added a compile-time warning for object_store.

@tustvold tustvold merged commit f812d2c into apache:master Oct 26, 2022
@ursabot
Copy link

ursabot commented Oct 26, 2022

Benchmark runs are scheduled for baseline = 4d5d10d and contender = f812d2c. f812d2c is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Skipped ⚠️ Benchmarking of arrow-rs-commits is not supported on ec2-t3-xlarge-us-east-2] ec2-t3-xlarge-us-east-2
[Skipped ⚠️ Benchmarking of arrow-rs-commits is not supported on test-mac-arm] test-mac-arm
[Skipped ⚠️ Benchmarking of arrow-rs-commits is not supported on ursa-i9-9960x] ursa-i9-9960x
[Skipped ⚠️ Benchmarking of arrow-rs-commits is not supported on ursa-thinkcentre-m75q] ursa-thinkcentre-m75q
Buildkite builds:
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java

@kylebarron
Copy link
Contributor

kylebarron commented Feb 21, 2023

and some of the compression features on parquet won't work

As a note on this, I've gotten zstd and lz4-flex to both compile to wasm, so it's possible to have better wasm parquet support if you're interested. For zstd I think it should work out of the box with arrow-rs IIRC (you may need to make sure you're building against the right clang, especially if on Mac, see https://github.com/kylebarron/parquet-wasm/pull/42/files). For lz4, we'd need to update arrow-rs to switch between lz4 and lz4-flex depending on whether building for wasm or not. See jorgecarleitao/parquet2#124 for the relevant code in parquet2. As of that, parquet2 builds to wasm32-unknown-unknown for all compressors. Ref also #180

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
object-store Object Store Interface parquet Changes to the parquet crate
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Parquet does not support wasm32-unknown-unknown target
4 participants