v0.22.0: Initial WebGPURenderer Support in the XR Blocks Simulator, On-Device AI Demos, Spatial UI Upgrades, Day/Night Lighting & Profiling

XR Blocks v0.22.0 Release Notes

We are excited to announce XR Blocks 0.22.0, focusing on initial WebGPURenderer support in the XR Blocks simulator, a wave of on-device AI demos (Gemma 4, EfficientSAM, super-resolution, offline translation), spatial UI upgrades (scroll views, editable text, resizable cards), simulator day/night lighting and profiling, and repository-wide toolchain modernization.

• Initial WebGPURenderer support in the XR Blocks simulator: THREE.WebGPURenderer now runs in the desktop simulator — the simulator compositor renders through an intermediate render target (SimulatorCompositor, unified Simulator.renderFrame), the reticle uses TSL NodeMaterial, and SimulatorCamera, the depth pipeline, and depth occlusion all accept WebGPURenderer, with the depthmap sample running on WebGPU. The spatial UI is not yet supported on WebGPURenderer — this is initial simulator support, not end-to-end coverage.
• On-Device AI Demos: An on-device Gemma 4 assistant with spatial scene context and markdown replies, on-device super-resolution, an EfficientSAM circle-to-search demo (plus the objects3d pipeline moved from SlimSAM to EfficientSAM-Ti on LiteRT), offline spatial captions & translation, interactive Teachable Machines ML, photo-to-3D reconstruction, and world-aligned metric depth from a single camera photo.
• Spatial UI: Scroll views and editable text fields, cards resized from their edge corners, a ported spatial-UI panels & gradient features gallery, icon SVG shapes merged into one mesh per fill color for fewer draw calls, proper font inheritance, objectFit honored for caller textures with the camera panel matching the feed aspect, and panel backface draw calls skipped while the camera is in front.
• Generative & Tracked Objects: Summon AI-generated objects onto the surfaces around you, ArUco tag tracking with objects3d binding and self-resolving device camera profiles, VRM occlusion fixes, and unofficial UV mapping with a Mona Lisa texture in the GNM demo.
• Simulator & Profiling: Manifest-driven, lazily-loaded day/night lighting, hands-mode navigation (Left Shift to navigate, right-click to look around), an instructions panel for the current simulator mode, a standalone Chrome profiler with HTML reports, and more reliable simulator startup and screenshot capture.
• Roomcraft Collaboration: Cross-device collaboration with shared motion, preserving reconnect state and incoming voice across sessions.
• AI Platform & Toolchain: Gemini calls migrated from generateContent to the Interactions API (SDK + all examples), the default Flash model updated to gemini-3.8-flash, the build migrated from Rollup to Rolldown, linting migrated from ESLint to Oxlint with TypeScript 7, three.js updated to r186, and 8 npm Dependabot security alerts resolved.
• Robustness Fixes: Correct top-origin depth UV conversion and depth-resource rebinding/release, stroke recording reset and degenerate-unistroke rejection, direct touch released when hand tracking is lost, hand tracking optional for XR entry, no orphaned live AI sessions, async-safe editor undo/redo, sprite raycasting crash fixes, and wrist-driven X/Z rotation support.

---

What's Changed

• Generative objects: summon AI-generated objects onto the surfaces around you by @salmanmkc in #405
• Object3d device fixes by @Peter-Pater in #487
• feat(layers): composition layer support behind enableLayers() by @salmanmkc in #516
• Raycaster Sprite on Context by @whuang37 in #555
• feat: added unofficial uv mapping to GNM demo and a mona lisa texture by @ruofeidu in #557
• Improved documentation of gnm and xrblocks templates. by @ruofeidu in #559
• chore: update default Gemini flash model to 3.8-flash by @dli7319 in #560
• Bump fast-uri from 3.1.5 to 3.1.7 in the npm_and_yarn group across 1 directory by @app/dependabot in #561
• Rename Emulator Scene to Daytime/Evening Loft. by @dli7319 in #562
• Bump the npm_and_yarn group across 2 directories with 4 updates by @app/dependabot in #564
• Update simulator hands mode with Left Shift navigation and Right Click Look around by @dli7319 in #565
• Show instructions for current simulator mode from settings panel by @dli7319 in #566
• fix: support wrist-driven X/Z rotation by @su-jin1425 in #567
• Bump @swc/html from 1.15.33 to 1.16.2 in /docs in the npm_and_yarn group across 1 directory by @app/dependabot in #568
• Bump svgo from 3.3.4 to 3.3.5 in /docs in the npm_and_yarn group across 1 directory by @app/dependabot in #570
• Bump the npm_and_yarn group across 2 directories with 2 updates by @app/dependabot in #572
• show XR entry failures and keep retries safe by @salmanmkc in #573
• refresh the headset camera before scripts run by @salmanmkc in #574
• release direct touch when hand tracking is lost by @salmanmkc in #575
• don't drive the simulator while a text field has focus by @salmanmkc in #576
• perf(context): skip redundant transform refreshes in semantic snapshots by @salmanmkc in #577
• reuse obstacle bounds after a rejected placement candidate by @salmanmkc in #578
• docs: remove redundant heading from PR template by @salmanmkc in #579
• Allow per-call object detection snapshots and backends by @salmanmkc in #580
• add a standalone Chrome profiler with HTML reports by @salmanmkc in #581
• feat(ui): add scroll views and editable text by @salmanmkc in #582
• feat(roomcraft): add cross-device collaboration and shared motion by @salmanmkc in #583
• fix(roomcraft): preserve reconnect state and incoming voice by @salmanmkc in #584
• fix(netblocks): move addon samples to demos so they build and run again by @salmanmkc in #585
• test: use jsdom storage in test workers by @salmanmkc in #587
• test(core): synchronize simulator startup without polling by @salmanmkc in #588
• feat(evals): add an output-adapted baseline comparison arm by @salmanmkc in #589
• Update to threejs r185 by @dli7319 in #590
• Update to threejs r186 by @dli7319 in #591
• Fix raycaster line and point threshold to 1cm in HitRegistry by @dli7319 in #592
• fix(depth): convert view UVs to top-origin before the depth transform by @salmanmkc in #593
• docs(samples): bring back the netblocks gallery entry by @salmanmkc in #594
• fix(deps): declare ws as an optional relay peer by @salmanmkc in #595
• fix(input): reset stroke recording on deactivate by @salmanmkc in #596
• docs: fix stale public-surface references by @salmanmkc in #597
• fix(input): reject degenerate unistroke input by @salmanmkc in #598
• fix(depth): release depth resources during core teardown by @salmanmkc in #599
• fix(depth): rebind the native depth texture every frame by @salmanmkc in #600
• test(input): pin stroke geometry and capture boundaries by @salmanmkc in #601
• test(interaction): cover reentrant direct-touch bounds queries by @salmanmkc in #602
• fix(demos): port broken demos to current sdk apis by @salmanmkc in #603
• fix(ai): prevent orphaned live sessions and accept image model names by @salmanmkc in #604
• fix(editor): preserve async undo and redo history by @salmanmkc in #605
• fix(core): settle screenshot failures and recover simulator startup by @salmanmkc in #606
• Add Phase 1 WebGPURenderer support and desktop simulator integration by @dli7319 in #607
• Add WebGPURenderer support for Reticle with TSL NodeMaterial by @dli7319 in #609
• Extract SimulatorCompositor and unify Simulator.renderFrame by @dli7319 in #611
• Vrm occlusion by @Peter-Pater in #612
• Update WebGPU simulator compositor to render to intermediate render t… by @dli7319 in #613
• Support WebGPURenderer in Simulator depth pipeline by @dli7319 in #614
• Persist pointerEvents and presentationKey in UIKitNodeBinding.commit by @dli7319 in #615
• Precompute unprojection rays and optimize grid vertex normals of DepthMesh by @dli7319 in #618
• feat: support WebGPURenderer in depthmap sample and simulator XREffects by @dli7319 in #620
• Add THREE.WebGPURenderer support to SimulatorCamera by @dli7319 in #622
• Update spatial ui move/resize text. by @dli7319 in #623
• Aruco Tag Tracking by @Peter-Pater in #624
• Add WebGPURenderer support for depth occlusion by @dli7319 in #625
• feat(ui): resize cards from their edge corners by @salmanmkc in #626
• chore(deps): resolve 8 npm Dependabot security alerts by @dli7319 in #628
• Aruco objects3d by @Peter-Pater in #631
• Docs drop aruco sample by @Peter-Pater in #632
• feat(demos): add on-device super-resolution demo by @salmanmkc in #633
• feat(demos): add on-device gemma 4 assistant demo by @salmanmkc in #634
• fix(core): make enabled hand tracking optional for xr entry by @salmanmkc in #636
• build: migrate from Rollup to Rolldown and TypeScript 7 by @dli7319 in #637
• Photo to 3d by @Peter-Pater in #638
• Feat/efficientsam demo by @dli7319 in #640
• feat(demos): add offline spatial captions and translation demo by @salmanmkc in #643
• Multiple Bug Fixes and UI Compositor by @whuang37 in #644
• Let ArucoTracker resolve the device camera profile itself instead of relying on object detection to do that by @Peter-Pater in #645
• demo(photo_depth): world-aligned metric depth from one camera photo by @Peter-Pater in #646
• chore: migrate from ESLint to Oxlint and upgrade to TypeScript 7 by @dli7319 in #647
• font inherits properly by @whuang37 in #648
• Fixed raycast crashes on sprite by @whuang37 in #649
• Interactive ML (Teachable Machines) by @whuang37 in #650
• perf(efficientsam): run full decoder on WebGPU and convert worker to module by @dli7319 in #651
• Add BEFORE_SESSION_START event and framebufferScaleFactor option by @dli7319 in #652
• build(deps): bump ip-address from 10.4.0 to 10.7.2 in the npm_and_yarn group across 1 directory by @app/dependabot in #653
• build(deps): bump the npm_and_yarn group across 2 directories with 3 updates by @app/dependabot in #655
• feat(objects3d): replace slimsam with efficientsam_ti on LiteRT by @dli7319 in #656
• feat(samples): port spatial UI panels and gradient features gallery by @dli7319 in #660
• perf(ui): merge icon SVG shapes into one mesh per fill color by @dli7319 in #662
• feat(simulator): manifest-driven, lazily-loaded day/night lighting by @dli7319 in #663
• feat(ai): migrate Gemini calls to the Interactions API by @dli7319 in #664
• fix(simulator): remove the instructions popup scrollbar by @dli7319 in #665
• feat(demos): migrate remaining example Gemini calls to the Interactions API by @dli7319 in #666
• ui: skip panel BackfaceLayer draw calls while the camera is in front by @dli7319 in #667
• build(deps): bump the npm_and_yarn group across 2 directories with 6 updates by @app/dependabot in #670
• build(deps-dev): bump @modelcontextprotocol/sdk from 1.30.0 to 1.32.1 in the npm_and_yarn group across 1 directory by @app/dependabot in #671
• fix(ui): honor objectFit for caller textures and match camera panel to feed aspect by @dli7319 in #676
• Bump to 0.22.0 by @dli7319 in #678

New Contributors

• @Peter-Pater made their first contribution in #487
• @su-jin1425 made their first contribution in #567

Full Changelog: v0.21.1...v0.22.0