Implementing the API (WebONNX) for doing inference on ONNX-serialized models on platform-available accelerators

[KOLANICH]

The idea is to implement a WebExtensions experiment that

• discovers the native libs for ML inference, available in the system, such as platform-provided frameworks like WinML, TF Lite, and Core ML, libs installed from packages, like onnxruntime (contains M$ telemetry!), libonnx and maybe even ONNX-mlir + LLVM 15
• binds to native libs available in the system
• provides WebExtensions with the API to infer models

More info: open-source-ideas/ideas#69

[KOLANICH]

The API should be designed so that almost exactly (but maybe under a different namespace, for example in webexts browser.onnx and in webpages as navigator.onnx) the same API could be exposed to web pages (allowing to use almost exactly the same code for them as within webexts).

[kpu]

You're absolutely right that native is faster. In fact the current WASM implementation is 10x slower than a proper native implementation, which could run in a sandbox. And that speed could also have been used to deliver better translation quality.

We're developing a native messaging extension here: https://github.com/jelmervdl/firefox-translations though it's still WIP.

Have you spoken to the Web Machine Learning group at W3C? https://www.w3.org/blog/2021/04/w3c-launches-the-web-machine-learning-working-group/ Their timeline was too long for us to use it.

[KOLANICH]

Have you spoken to the Web Machine Learning group at W3C?

No, but I guess I should. Thank you for mentioning it.

[marco-c]

We are not going to make any improvements or fixes to the addon since we are now focusing on the built-in version.
Once some API along these lines is available in Firefox, we'll try to make use of it to speed up model inference.