[Question] Rust SDK + WebAssembly + GPU?

[louis030195]

❓ General Questions

maybe dumb question

in theory, is it possible to build a lib with mlc-llm Rust SDK, compile to WebAssembly, and use in the web with a GPU?

or even use in https://github.com/tauri-apps/tauri

?

[tqchen]

try out https://webllm.mlc.ai/, which already works in web

[louis030195]

@tqchen ? this does not answer my question, I'm already aware of this and used it already

what is the rust sdk for? what people do with?

thanks!

[tqchen]

OK I get what you mean. As of now the rust SDK does not yet compiles to wasm/webgpu. It can be used to interact with the chat module in other gpus(where the gpu runtime was compiled and supported through tvm runtime)

The main technical setup that is needed for webgpu is a webgpu runtime, regardless of language, we need a way to access webgpu. As of now we leverage javascript runtime with TVM FFI to integrate webgpu which is most robust route. That means the wasm code needs to work together with a JS runtime.

In theory we can make tvm target webgpu native instead, in which case there might be some possibilities. We do not yet bandwidth to go this routine

[tqchen]

closing as the original question was answered and there is no extra actionable items

[louis030195]

@tqchen I'm going to try the Rust SDK, will use my mac m3 gpu, but if it goes well i am interested in the future in compiling my project to WASM and run on the browser

how hard is it to refactor rust sdk to compile to wasm + webgpu (and any gpu, say i use the wasm on an iphone ...) (not super familiar with wasm yet) ?

any basic instructions and i can send a PR to implement this when time is right

[louis030195]

PS: seeing the rust sdk has been removed, will copy paste the code in my project np

[tqchen]

We removed rust part for now because we are transitioning to the new MLCEngine backend and the original rust was for legacy ChatModule, @LeshengJin is looking into adding support

Likely you can use rust plus m3. for wasm+ webgpu, i think it is possible but might need some working together with the emcc runtime