[WIP] 🚀🚀🚀 Transformers.js V3 🚀🚀🚀

[xenova]

In preparation for Transformers.js v3, I'm compiling a list of issues/features which will be fixed/included in the release.

• WebGPU support (upgrade onnxruntime-web to 1.17.0).
  • closes WebGPU Support #20
  • closes [Feature request] WebGPU support #73
  • closes [Bug] image segmentation very slow #79
  • closes Whisper on webGPU? #100
  • closes [Question] A WebGPU-accelerated ONNX inference run-time #119
  • closes [Feature request] Update ONNX runtime to version 1.15.1 #298
  • closes GPU Acceleration to increase performance #377
  • closes [Feature request] Real time whisper transcription #405
  • closes How do I use WebGL as executionProvider? #505
  • closes YOLOS model extremely slow #533
• Fix logging/disable warnings (onnxruntime-web → 1.17.0). Closes:
  • closes [Question] How to stop warning log #270
  • closes env.backends.onnx.logLevel = "fatal"; does not work #529
• Fix WASM backend for large models (onnxruntime-web → 1.17.0). Closes:
  • closes An error occurred during model execution: "RangeError: offset is out of bounds". #499
• Deno support (upgrade sharp.js to 0.33.x). Closes:
  • closes [Feature request] Deno Support #78
  • closes Sharpe Linux-x86 #541
• CommonJS compatibility. Closes [Feature request] CommonJS compatability #152
• Skip local model check when running in-browser, unless explicitly set by user. This is an issue experienced by many beginners, where requests made to localhost redirect to error page, but dev server incorrectly returns status code 200. Closes
  • closes [Question] React.js serve sentence bert in browser keep reporting models not found. #328
  • closes getModelJSON fails with a bundler (see reproduction) #366
• Improve unit test suite and allow local testing. Closes Running tests locally fail #491
• Create model conversion space. Closes
  • closes [Visual Model Conversion] Do-It-Yourself: A visual tool that allows users to convert their own models without sweats and tears #222
  • closes [Question] How to convert to ONNX a fine-tuned model #274
• Upgrade conversion script dependency versions (+fixes sentence-transformers conversions). Closes
  • closes [Question] distiluse-base-multilingual-cased-v2 - wrong vector dimension (768 vs 512) in onnx version? #230
• Versioned documentation, so that users still on v2 will be able to access the correct documentation.
• Improve PretrainedModel, PretrainedTokenizer, and Processor types. In a similar way to how the pipeline API has conditional types, we'll add the same for the other classes accessible by users.
• Consistency issues:
  • topk -> top_k parameter.
  • Tensor transpose -> permute
  • Like the python library, encapsulate multimodal preprocessing in the AutoProcessor class, which uses image processor and tokenizer
• Improved model caching of revisions, and purge cache if model files are out-of-date. Ideally, this should be enabled by a setting to avoid unnecessary network calls to check versions if the files are present in cache.
• Improve pipeline fallback errors
  • closes Unsupported model type: whisper and CORS error #314

How to use WebGPU

First, install the development branch

npm install xenova/transformers.js#v3

Then specify the device parameter when loading the model. Here's example code to get started. Please note that this is still a WORK IN PROGRESS, so the following usage may change before release.

import { pipeline } from '@xenova/transformers';

// Create feature extraction pipeline
const extractor = await pipeline('feature-extraction', 'Xenova/all-MiniLM-L6-v2', {
    device: 'webgpu',
    dtype: 'fp32', // or 'fp16'
});

// Generate embeddings
const sentences = ['That is a happy person', 'That is a very happy person'];
const output = await extractor(sentences, { pooling: 'mean', normalize: true });
console.log(output.tolist());

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[Huguet57]

Hey! This is great. Is this already in alpha?

[young-developer]

@xenova There is new version 1.17.3 of onnxruntime-web. I tested with wav2vec and there is new error so looks like a progress 😄

[xenova]

Segment Anything Encoder now works with WebGPU: up to 8x faster! (online demo)

sam-webgpu.mp4