v0.2.0 💨

ruNNtime zoo v0.2.0: EmbeddingGemma 2, GPU in GPU out, and a lot faster 🏃

Six weeks after v0.1.0, every model got faster, EmbeddingGemma 2 embeds text, images, video and audio into one space, and depth estimation can run from a video frame into a canvas without touching the CPU.

const embedder = await createEmbedder(models.embedding.EMBEDDINGGEMMA_2.DEFAULT);
const query = await embedder.embed('cats asleep on a sofa', { task: 'query' });
const photo = await embedder.embed({ image: catsPhoto });
similarity(query, photo); // 0.76

Performance

We ported a set of kernels from Hugging Face's webgpu-kernels templates into TypeGPU 🤗
All numbers were measured on the same MacBook Pro (M2 Pro), v0.1.0 vs v0.2.0.

Workload	v0.1.0	v0.2.0	ONNX Runtime via transformers.js
Moonshine tiny, speech to text	89× realtime	137× realtime	20×
Privacy filter	32 texts/s	61 texts/s	14 texts/s
YOLO26 m, object detection	13.6 FPS	25.1 FPS	–
YOLO26 m, instance segmentation	7.2 FPS	16.4 FPS	–
DepthART, depth estimation	51 FPS	74 FPS	–
all-MiniLM-L6-v2, batch of 32	2.7 ms/paragraph	1.6 ms/paragraph	7.9 ms

If you want to quickly check out how runntime can help accelerate performance of your existing transformers.js code, see the docs.

EmbeddingGemma 2

createEmbedder runs Google's EmbeddingGemma 2: text in 100+ languages, images, video and audio into one 768-dim space.

EmbeddingGemma 2	ruNNtime	ONNX Runtime via transformers.js
Text, batch of 32	40.5 paragraphs/s	10.2 paragraphs/s
Images	1.73 images/s	1.52 images/s
Audio	69.5× realtime	61.9× realtime

Docs: Embedding.

GPU in, GPU out depth estimation

encodeEstimateDepth reads an image that is already on the GPU (a GPUTexture, a TypeGPU texture, or a GPUExternalTexture from a <video>) and writes the depth map into a texture or buffer you own. Nothing is read back. This means that you can do inference with zero copies between the CPU and the GPU 🚀! Thanks @iwoplaza for your help on this.

function frame() {
  estimator.encodeEstimateDepth({
    source: root.device.importExternalTexture({ source: video }),
    target: context.getCurrentTexture(),
    normalize: 'minmax',
  });
  video.requestVideoFrameCallback(frame);
}

DepthART itself went from 51 to 74 FPS (19.3 to 13.3 ms per image). Docs: Depth estimation, GPU in, GPU out.

Also in this release

• Error codes. Every error ruNNtime throws is a RunntimeError with a code (LOAD_FAILED, UNSUPPORTED_DEVICE, INVALID_ARGUMENT, ...). isRunntimeError(err, code) narrows it. See Error handling.
• GPU memory stays flat. Models free what they allocate, and inputs of varying size no longer pile up buffers.
• transformers.js plugin. pipeline('depth-estimation') with the Depth Anything V2 model ids now runs DepthART on ruNNtime.
• Tokenizers. MiniLM and Moonshine now use the standard Hugging Face tokenizers instead of our own ports.
• Live speech to text no longer flashes looping previews on audio cut mid-word.
• Benchmarks page now covers depth, detection, pose, segmentation, the privacy filter and EmbeddingGemma 2.

Breaking changes

createTextEmbedder and models.textEmbedding are replaced by createEmbedder and models.embedding. The model is read from the weights, so all-MiniLM-L6-v2 users only change the names:

- const embedder = await createTextEmbedder(models.textEmbedding.MINILM.DEFAULT);
+ const embedder = await createEmbedder(models.embedding.MINILM.DEFAULT);

Model coverage

Task	Factory	Models
Embedding	createEmbedder	EmbeddingGemma 2 (text, image, video, audio), all-MiniLM-L6-v2
Speech to text	createSpeechToText	Moonshine tiny, base, streaming tiny
Privacy filter	createPrivacyFilter	OpenAI privacy-filter, int8
Object detection	createObjectDetector	YOLO26 n, s, m (COCO-80)
Instance segmentation	createInstanceSegmenter	YOLO26-seg n, s, m
Pose and keypoints	createKeypointDetector	YOLO26-pose n, s, m (17 landmarks)
Depth estimation	createDepthEstimator	DepthART s, b, CPU or GPU in and out
Image classification	createImageClassifier	MobileNetV4-Conv-S (ImageNet-1k)