WebGPU/CoreML support for node.js

[nemphys]

Feature request

Now that onnxruntime-node officially supports WebGPU on all platforms and CoreML on macOS (see https://github.com/microsoft/onnxruntime/tree/main/js/node#requirements), it would be nice to include them in transformers.js, too.

Motivation

Right now, all platforms except linux lack GPU EP support, missing out on the apparent performance gains.

Your contribution

I can submit a PR with the required changes, but it involves version bumping of onnxruntime to (at least) 1.22.0.

[nemphys]

@xenova say the word and I will send the PR 😄

[xenova]

Hi there! 👋 This is indeed being worked on in #1382 😄 The Node.js WebGPU & CoreML integration is already working there, in fact!

[nemphys]

Wow nice! I hand't noticed the V4 effort, guess I will have to go with a patched V3 until this is released.

I tried using WebGPU on my M2 Max MacbookPro and it actually works fine 😃.

The only thing that made me wonder is that, although the models I tried indeed seem to be executed solely on GPU, I didn't see any performance gains; inference times where mostly the same with (or even a little slower than) CPU 🤔

[xenova]

guess I will have to go with a patched V3 until this is released.

You should be able to install directly from github; something like

npm install https://github.com/huggingface/transformers.js/tree/v4

The only thing that made me wonder is that, although the models I tried indeed seem to be executed solely on GPU, I didn't see any performance gains; inference times where mostly the same with (or even a little slower than) CPU 🤔

Can you give some more info? 👀 any models in particular? Also, was this tested with transformers.js v4, or with your patched v3?

[nemphys]

My initial tests were with the patched v3 version (using onnxruntime-node 1.23), but I just tested with 4.0.0-alpha.0 and got similar results:

Xenova/clip-vit-base-patch32 (fp32): CPU ~20ms, WebGPU ~16ms <---- small gain)
Xenova/bge-m3 (fp32): CPU ~30ms, WebGPU ~37ms <---- worse)
Xenova/siglip-base-patch16-224 (fp32): CPU ~70ms, WebGPU ~21ms <---- this is actually good)

[nemphys]

@xenova is there any new optimization that needs to be done to existing models for more efficient GPU inference?

I noticed task called "Migrate certain ONNX models on the HF hub to be v3 & v4 compatible (mainly focused on older models)." in the V4 page and it made me wonder.