Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature: Investigate WASM/WASI targets #26

Open
sjrusso8 opened this issue Apr 30, 2024 · 1 comment
Open

Feature: Investigate WASM/WASI targets #26

sjrusso8 opened this issue Apr 30, 2024 · 1 comment
Labels
enhancement New feature or request help wanted Extra attention is needed

Comments

@sjrusso8
Copy link
Owner

sjrusso8 commented Apr 30, 2024

Description

Being able to compile the rust bindings into wasm32-unknown-unknown and/or wasm32-wasi would be interesting. This could allow some interesting interactions between a browser and spark. A wasm32-wasi target would allow spark programs to run on any runtime.

Early Experiments

A feature flag under the core bindings for wasm already exists and does compile successfully to those targets mentioned above. The issue arises when trying to send a HTTP1.1 request with grpc-web to the Spark Connect server, which only accepts normal HTTP2 grpc requests. There are methods of standing up a proxy server with envoy to forward the gRPC browser request to the backend server. But this feels like a lot of effort for the client to do.

The branch feat/wasm contains the early experiment and trying to run wasm with wasmtime. Issue arises with using async code in wasm. There is probably a way to code it correctly, but I don't have time to finish the experiment

@sjrusso8 sjrusso8 added enhancement New feature or request help wanted Extra attention is needed labels Apr 30, 2024
@sjrusso8
Copy link
Owner Author

sjrusso8 commented May 7, 2024

Was able to create a wasm program to run a simple spark structure streaming job. The brach feat/wasm was updated with envoy to handle the grpc-web requests. Might consider having an 'example' section for how to setup envoy and use the wasm feature flag.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request help wanted Extra attention is needed
Projects
None yet
Development

No branches or pull requests

1 participant