Skip to content

v5.3.12 - Text & Image to 3D in your browser (TRELLIS.2 on WebGPU)

Choose a tag to compare

View all tags
@LostBeard LostBeard released this 17 Oct 22:56
· 216 commits to master since this release
v5.3.12
c07e1ae

SpawnDev.ILGPU.ML 5.3.12

Type a prompt or drop a photo, get a textured 3D model. Entirely on your own GPU, in your browser.
No server, no upload, no install. Microsoft's TRELLIS.2 (4B parameters) running on WebGPU from C#.

Text to 3D - a treasure chest generated in the browser

🧊 Try it: lostbeard.github.io/SpawnDev.ILGPU.ML/text-to-3d (photo mode: /image-to-3d)
📦 NuGet: SpawnDev.ILGPU.ML 5.3.12 (requires SpawnDev.ILGPU 5.3.8)
📖 How it works: Docs/trellis2.md
💚 Sponsor: github.com/sponsors/LostBeard

Text & Image to 3D, 100% client side

To our knowledge this is the first fully client-side TRELLIS.2 on the web. Every stage runs on the visitor's GPU over WebGPU:

  1. SD Turbo draws the object from your prompt (text mode)
  2. RMBG-1.4 cuts it out
  3. TRELLIS.2-4B builds it: DINOv3 image features, three 1.3B-parameter flow transformers (sparse structure, shape, texture), sparse VAE decoders, a dual-grid mesh
  4. Mesh clustering + simplification, UV atlas, GPU texture bake
  5. Out comes a PBR GLB (base color + metallic-roughness), previewed in the page and ready to download

The page shows live stage timings, a rotating preview and a GLB download button.

MEASURED on an RTX 4070 in Chrome (AOT build, weights cached):

time
Photo -> GLB, fresh page 148 s
Photo -> GLB, next generation 134 s
Prompt -> GLB 180 s

The first visit also downloads about 10 GB of weights (opt-in, cached in browser storage), so that run is bound by your connection.

What it took to fit a 4B-parameter 3D model into a browser tab

  • Portable WebGPU kernels written in C#: a register-tiled GEMM (128x128 tiles, one kernel for every weight and row-gather variant) and flash attention (8 lanes x 4 queries per thread, with a shared-memory fallback for devices without subgroups). Transpiled to WGSL by SpawnDev.ILGPU.
  • 10 GB of checkpoints vs. a browser quota: RangeStreamModelSource.PlanCacheAsync caches as many whole files in OPFS as the quota allows and streams the rest with pipelined HTTP range requests (4 in flight) straight to the GPU, without passing through the .NET heap.
  • 3M-triangle meshes: MeshClusterer does adaptive octree quadric clustering on the device before the edge-collapse simplifier.
  • Shader compile time: WebGPUAccelerator.PrecompileAsync (new in SpawnDev.ILGPU 5.3.8) builds the large kernels' pipelines in parallel in the background while the weights load, instead of serially at first launch.
  • Blazor WebAssembly AOT for the demo: mesh simplification 41.3 s -> 2.6 s, UV atlas 16.8 s -> 0.8 s.

On the desktop the same library runs TRELLIS.2 on CUDA (tensor cores via inline PTX), OpenCL and CPU: about 45-65 s per model on an RTX 4070 (12 GB), with an optional 1024 pipeline.

Use it from C#

using SpawnDev.ILGPU.ML.Trellis2;

// Weights from the hub: cached where they fit, streamed otherwise
var ranged = new RangeStreamModelSource(hubModelSource);
await ranged.PlanCacheAsync(Trellis2Weights.HubWeightFiles(), freeQuotaBytes - 300_000_000L);

using var gen = new Trellis2ImageTo3D(accelerator, Trellis2Weights.FromModelSource(ranged));
byte[] glb = await gen.GenerateGlbAsync(rgbaPixels, width, height);   // RGBA, alpha = foreground

Trellis2TextTo3D does prompt -> image -> cut-out -> GLB in one call.

Also in this release line (5.3.9 - 5.3.12)

  • Gemma multi-turn chat fixed (5.3.12): the SentencePiece tokenizer now follows the GGUF tokenizer.ggml.add_space_prefix flag. Gemma 2/3 chats no longer pick up a stray space token at every turn boundary; greedy output matches Ollama exactly. The /ai-chat demo now lists the instruction-tuned Gemma 3 270M.
  • Face detection + recognition (5.3.11): YuNetFaceDetector + SFaceRecognizer, the pair OpenCV's FaceDetectorYN / FaceRecognizerSF use, checked against OpenCV.
  • Streaming speech to text (5.3.10): StreamingTranscriber (NVIDIA Nemotron Speech Streaming 0.6B) gives words as the audio arrives in 160 ms chunks. LibriSpeech test-clean WER 5.63% (sherpa-onnx on CPU: 6.17%).
  • Speech pipelines (5.3.9): Parakeet TDT 0.6B v3 speech to text, speaker embeddings, Smart Turn end-of-turn detection, each matched to onnxruntime.
  • Text to Speech is now verified in the demo (Kokoro + ZipVoice voice cloning).

Full details: CHANGELOG.md

Licences

TRELLIS.2 code and weights: MIT. DINOv3 weights: Meta's DINOv3 licence ("Built with DINOv3"). SD Turbo: Stability AI's licence. RMBG-1.4: BRIA's licence. No weights ship in the package; the demo downloads them from the Hugging Face hub on first use. Check each licence before commercial use.

Assets 2
Loading