Skip to content

perf(utils): applyBVH helper for opting scenes into accelerated raycasts - #372

Merged
ruofeidu merged 17 commits into
google:mainfrom
salmanmkc:perf/bvh-raycast-sdk
Jun 16, 2026
Merged

perf(utils): applyBVH helper for opting scenes into accelerated raycasts#372
ruofeidu merged 17 commits into
google:mainfrom
salmanmkc:perf/bvh-raycast-sdk

Conversation

@salmanmkc

@salmanmkc salmanmkc commented Jun 15, 2026

Copy link
Copy Markdown
Contributor

the SDK's per-frame interaction raycaster (Input.performRaycastOnScene) walks every triangle of every mesh under xb.core.scene via THREE.Raycaster.intersectObject (recursive). for demos with non-trivial scene geometry the per-triangle walk dominates the main thread. surfaced in a portals trace where the call chain intersectTriangle -> _computeIntersections -> raycast -> xb.intersect accounted for ~550 ms over a 12 s window.

three-mesh-bvh's accelerated raycaster solves this but has to be opted into per geometry. two public helpers added in src/utils/BVHRaycast:

  • enableAcceleratedRaycast(): installs the THREE.Mesh prototype patch + BufferGeometry.computeBoundsTree helper. idempotent across callers so multiple subsystems can ping it safely.
  • applyBVH(root, {recursive?}): walks the tree and builds a bounds tree on every mesh's geometry. demos opt in by calling this on the root of their scene after meshes are added.

three-mesh-bvh is loaded as an optional dynamic import (devDependency, type-only build import, await import('three-mesh-bvh') with try/catch). apps without it in their importmap don't break, they just don't get the accelerated raycast. both helpers are async and resolve to false in that case. same pattern as troika-three-text in TextView.

applyBVH skips SkinnedMesh (animated vertices invalidate the bind-pose bvh, and three's SkinnedMesh.raycast overrides the patched prototype anyway) and BatchedMesh (needs three-mesh-bvh's separate computeBatchedBoundsTree building per-draw-range trees on this.boundsTrees). InstancedMesh is kept in, its per-instance _mesh.raycast() does route through the patched Mesh.prototype so the bvh on the shared geometry accelerates each instance test.

geometries without a bounds tree continue using the stock raycaster, so flipping the prototype patch on globally is safe for existing code that never calls applyBVH.

opted in demos/portals by calling xb.applyBVH(this) at the end of PortalGalleryScene.init() after all 5 immersive scenes are added.

measured impact

portals demo trace before vs after (this PR + #371 both applied):

function original portals trace this trace change
intersectTriangle 167 ms 34 ms -80%
getVertexPosition 137 ms 50 ms -64%
checkGeometryIntersection 178 ms 12 ms -93%
_computeIntersections 68 ms 16 ms -77%
WebGLRenderer.setSize 154 ms 1 ms -99% (from #371)
getProgramInfoLog 124 ms 49 ms -60% (incidental, prob fewer shader compiles)

idle went from 8% -> 60% of the trace window. massive frame-budget recovery on a demo that does interaction raycasting + occlusion.

12 tests cover the helpers: prototype patch install / idempotency, recursive build + recursive=false, dispose, per-subclass skip filter (SkinnedMesh, BatchedMesh) + InstancedMesh build, a mixed-tree filter test (all 4 subclass types in one applyBVH call, locks the policy in), end-to-end raycast through the patched path, and cold-dispose safety. 310 tests total.

related: #367 (BVH for face_mirror's per-landmark raycasts; refactored to depend on this PR's util). #371 (occlusion pass setSize fix, contributes the setSize row above).

…casts

The SDK's per-frame interaction raycaster walks every triangle of
every mesh under xb.core.scene via THREE.Raycaster.intersectObject
(recursive). For demos with non-trivial scene geometry (e.g. portals
with 5 loaded immersive worlds), the per-triangle walk dominates the
main thread.

three-mesh-bvh's accelerated raycaster solves this but it has to be
opted into per geometry. Adds two public helpers:

- enableAcceleratedRaycast(): installs the THREE.Mesh prototype patch
  and BufferGeometry helpers. Idempotent across callers (so multiple
  subsystems can ping it).
- applyBVH(root, {recursive?}): walks the tree and builds a bounds
  tree on every mesh's geometry. Demos opt in by calling this on the
  root of their scene after meshes are added.

Geometries without a bounds tree continue to use the stock raycaster,
so flipping the prototype patch on globally is safe for existing code
that never calls applyBVH.

Also adds three-mesh-bvh to package.json + rollup externals so it
ships consistently across the SDK and addons. 6 new tests cover
prototype patch install, idempotency, recursive build, no-rebuild,
recursive=false stop, and dispose.
Portals demo loads 5 immersive scenes into xb.core.scene. The SDK's
per-frame interaction raycast (Input.performRaycastOnScene) walks
every triangle under the scene root each controller frame, and the
per-triangle walk dominated the main thread in a perf trace before
this.

Calls xb.applyBVH(this) at the end of PortalGalleryScene.init() so
the bounds tree is built once per geometry after all immersives are
added. Subsequent raycasts go through three-mesh-bvh's accelerated
path (O(log triangles) per ray instead of O(triangles)).

Adds three-mesh-bvh to the demo importmap. Same version pinned by
the SDK and other demos.
salmanmkc added a commit to salmanmkc/xrblocks that referenced this pull request Jun 15, 2026
Drop the inline three-mesh-bvh prototype patches and import
enableAcceleratedRaycast() from src/utils/BVHRaycast (added in google#372).
The helper is idempotent across modules so other subsystems that
also install BVH (objects_3d, portals) share the same patch without
fighting over the prototype.

This PR's branch is stacked on google#372 so the util is available; merge
google#372 first.
@salmanmkc salmanmkc changed the title utils: applyBVH helper for opting scenes into accelerated raycasts perf(utils): applyBVH helper for opting scenes into accelerated raycasts Jun 15, 2026
per review feedback on google#367, three-mesh-bvh should be optional like
troika-three-text is for TextView: type-only import at build time,
dynamic import at runtime with try / catch + status tracking. apps
without three-mesh-bvh installed (or without it in their importmap)
keep working with the stock three.js raycaster.

changes:

- moved three-mesh-bvh from dependencies to devDependencies (still
  needed for the SDK's own build + tests, but not forced on consumers)
- import type * as BVH from 'three-mesh-bvh' for compile-time types
- enableAcceleratedRaycast() and applyBVH() are now async; they kick
  off the dynamic import on first call and share the same promise on
  subsequent calls
- on import failure, both helpers log a one-line warn and return
  false (or no-op for applyBVH). raycasts fall back to the stock
  walker
- new isBVHReady() sync check for consumers that want to know whether
  the patches are installed
- disposeBVH() stays sync; no-op when BVH was never loaded

tests updated to await the async helpers. 295 tests still passing.

mirrors the dynamic-import pattern in src/ui/components/TextView.ts
for troika-three-text.
salmanmkc added a commit to salmanmkc/xrblocks that referenced this pull request Jun 15, 2026
Now that google#372's enableAcceleratedRaycast is async and three-mesh-bvh
is an optional dynamic import, the per-detection computeBoundsTree
call in getDepthMeshSnapshot needs to gate on whether the BVH module
has actually loaded. When BVH isn't available (no install, no
importmap), we skip the bounds-tree build and the stock raycaster
handles the per-landmark intersections.

Functionally a no-op for any consumer that already has three-mesh-bvh
in their importmap (face_mirror demo).
dli7319 pushed a commit to salmanmkc/xrblocks that referenced this pull request Jun 15, 2026
Drop the inline three-mesh-bvh prototype patches and import
enableAcceleratedRaycast() from src/utils/BVHRaycast (added in google#372).
The helper is idempotent across modules so other subsystems that
also install BVH (objects_3d, portals) share the same patch without
fighting over the prototype.

This PR's branch is stacked on google#372 so the util is available; merge
google#372 first.
dli7319 pushed a commit to salmanmkc/xrblocks that referenced this pull request Jun 15, 2026
Now that google#372's enableAcceleratedRaycast is async and three-mesh-bvh
is an optional dynamic import, the per-detection computeBoundsTree
call in getDepthMeshSnapshot needs to gate on whether the BVH module
has actually loaded. When BVH isn't available (no install, no
importmap), we skip the bounds-tree build and the stock raycaster
handles the per-landmark intersections.

Functionally a no-op for any consumer that already has three-mesh-bvh
in their importmap (face_mirror demo).
@ruofeidu

Copy link
Copy Markdown
Collaborator

Awesome!

We discussed BVH years ago in XR Blocks before releasing, but didn't end up with external demos, since while three-mesh-bvh accelerates raycasting from O(n) to O(log n), it is not a silver bullet and comes with significant trade-offs in buildling the tree & memory overhead. And it may break dynamic geometry. The BVH assumes the vertices are static. For simple UI panels, basic primitives, or low-poly elements, the stock linear triangle traversal is so fast that the overhead of traversing a tree structure actually makes the BVH slower (not to mention the wasted build time). But it's a good thing to add on! But it should absolutely NOT be enabled globally.

BVH should only be applied to dense, static environmental meshes (like the Portals demo with multiple loaded immersive worlds, or static 3DGS photogrammetry scans). For simple UI, dynamic objects, and characters, the stock three.js raycaster is both safer and more efficient.


Some issues found by Gemini:

Minor Issue to Consider (The SkinnedMesh Trap):
In src/utils/BVHRaycast.ts, applyBVH does the following:

if (obj instanceof THREE.Mesh && obj.geometry) {
  // ... compute bounds tree ...
}

Because THREE.SkinnedMesh and THREE.InstancedMesh inherit from THREE.Mesh, the instanceof check evaluates to true. This means applyBVH will spend CPU time and memory generating a bounds tree on them. However, SkinnedMesh and InstancedMesh override .raycast() on their own prototypes in three.js. Since enableAcceleratedRaycast() only patches THREE.Mesh.prototype.raycast, the BVH acceleration will actually be ignored for them, falling back to stock raycasting anyway.

Recommendation: You may want to skip computing the bounds tree for them to save time and memory:

if (
  obj instanceof THREE.Mesh && 
  !(obj instanceof THREE.SkinnedMesh) && 
  !(obj instanceof THREE.InstancedMesh) &&
  obj.geometry
) {

@ruofeidu
ruofeidu self-requested a review June 15, 2026 21:43
@ruofeidu ruofeidu self-assigned this Jun 15, 2026
@ruofeidu ruofeidu added the algorithm spatial algorithm label Jun 15, 2026
@salmanmkc salmanmkc closed this Jun 15, 2026
@salmanmkc salmanmkc reopened this Jun 15, 2026
salmanmkc added a commit to salmanmkc/xrblocks that referenced this pull request Jun 15, 2026
per review feedback on google#372, BVH isn't a silver bullet: tree build
+ memory cost, assumes static vertices, and the traversal can be
slower than walking a few triangles for low-poly meshes. should only
be applied to dense static environmental meshes, not globally.

three changes:

- applyBVH now skips THREE.SkinnedMesh and THREE.InstancedMesh. both
  override .raycast() on their own prototypes, so the BVH would never
  be consulted for them anyway. previously we'd pay the build cost
  for nothing.

- applyBVH docstring now spells out the when/when-not. flags the
  dense-static rule and warns against global application.

- demos/portals scopes the applyBVH call to each immersive sub-tree
  instead of the whole gallery root. the gallery root also holds
  portal discs, floating labels, the fade sphere, etc. which are
  low-poly / dynamic and don't benefit.

2 new tests cover the SkinnedMesh + InstancedMesh skip. 306 tests
total (was 304).
Both subclasses override .raycast() on their own prototypes (not on
THREE.Mesh.prototype), so the BVH path is never consulted for them.
Building the bounds tree on them just wastes CPU + memory.

2 new tests cover the skip.
Calls out the dense-static rule + warns against global application.
Useful guidance for future demo authors so the helper doesn't get
sprayed at every scene root.
Previously called xb.applyBVH on the entire gallery root which also
holds portal discs, floating labels, the fade sphere, etc. Those are
low-poly / dynamic and don't benefit from BVH. Apply per immersive
scene instead so we only pay the build cost on the dense static
geometry the BVH is actually meant for.
@salmanmkc
salmanmkc force-pushed the perf/bvh-raycast-sdk branch from d2987a9 to 942d971 Compare June 15, 2026 22:14
@salmanmkc
salmanmkc marked this pull request as draft June 15, 2026 22:24
Each immersive sub-tree contains exactly one shader skybox sphere with
`raycast = () => {}`, so the loop was building bounds trees no
raycaster ever consults: pure CPU/memory cost. Once an immersive gains
real raycastable geometry (loaded models, etc.), opt in again on those
specific meshes.
The original skip rationale ("InstancedMesh overrides .raycast() on its
own prototype so the BVH would never be consulted") was incorrect.

InstancedMesh.prototype.raycast (three.js v0.184.0,
src/objects/InstancedMesh.js:269-303) sets a shared internal
`_mesh.geometry = this.geometry` and calls `_mesh.raycast()` once per
instance. That call routes through the patched `Mesh.prototype.raycast`
(acceleratedRaycast), which short-circuits to `geometry.boundsTree`
whenever the shared geometry has one. So building a BVH on the shared
geometry accelerates every per-instance ray test.

Drops the InstancedMesh skip + the lock-in test, replaces with a test
that confirms the boundsTree IS built.
`THREE.BatchedMesh` is a `Mesh` subclass with its own raycast override
(three.js v0.184.0, src/objects/BatchedMesh.js:1382-1442) that iterates
per draw range and adjusts an internal mesh's drawRange before each
ray test.

three-mesh-bvh ships a dedicated pair for this case
(src/utils/ExtensionUtilities.js):
  - `computeBatchedBoundsTree(geometryId)` builds per-draw-range BVHs
    stored on `this.boundsTrees` (plural) on the BatchedMesh itself
  - `acceleratedRaycast` has a separate `if (this.isBatchedMesh) ...`
    branch that consults `boundsTrees` keyed by `geometryId`

Calling the standard `computeBoundsTree()` on the BatchedMesh's combined
geometry would index the full batched buffer (not per-draw-range), and
`acceleratedRaycast`'s BatchedMesh branch wouldn't find `boundsTrees`
either way. The standard helper is wrong for BatchedMesh.

Conservative skip until apps that need it call the batched helpers
directly. Doc + test included.
Single applyBVH call on a Group containing a regular Mesh + SkinnedMesh
+ InstancedMesh + BatchedMesh + plain Object3D, asserting each gets the
right treatment in one pass:
  - Mesh         → boundsTree built
  - SkinnedMesh  → skipped
  - InstancedMesh → boundsTree built (per-instance raycast uses it)
  - BatchedMesh  → skipped
  - Object3D     → not a mesh, naturally untouched

Locks the per-subclass filter behavior in as a single end-to-end test
rather than relying solely on the isolated per-subclass tests above.
End-to-end raycast: build BVH on a PlaneGeometry mesh, fire
raycaster.intersectObject through the patched Mesh.prototype.raycast,
assert the hit data is correct. Catches regressions where the
prototype patch silently breaks raycast results (e.g. if
three-mesh-bvh's acceleratedRaycast signature ever changes, or the
BufferGeometry helper hookup drifts).

Cold-dispose safety: vi.resetModules() to force bvhProtoPatched back
to false, then call disposeBVH(root) on a tree that never had
applyBVH applied. Asserts no throw and no boundsTree side-effect.
Exercises the early-return path on a cold module that the existing
tests can't hit because module state persists across them.
@salmanmkc
salmanmkc marked this pull request as ready for review June 15, 2026 23:17
@salmanmkc

Copy link
Copy Markdown
Contributor Author

Awesome!

We discussed BVH years ago in XR Blocks before releasing, but didn't end up with external demos, since while three-mesh-bvh accelerates raycasting from O(n) to O(log n), it is not a silver bullet and comes with significant trade-offs in buildling the tree & memory overhead. And it may break dynamic geometry. The BVH assumes the vertices are static. For simple UI panels, basic primitives, or low-poly elements, the stock linear triangle traversal is so fast that the overhead of traversing a tree structure actually makes the BVH slower (not to mention the wasted build time). But it's a good thing to add on! But it should absolutely NOT be enabled globally.

BVH should only be applied to dense, static environmental meshes (like the Portals demo with multiple loaded immersive worlds, or static 3DGS photogrammetry scans). For simple UI, dynamic objects, and characters, the stock three.js raycaster is both safer and more efficient.

Some issues found by Gemini:

Minor Issue to Consider (The SkinnedMesh Trap): In src/utils/BVHRaycast.ts, applyBVH does the following:

if (obj instanceof THREE.Mesh && obj.geometry) {
  // ... compute bounds tree ...
}

Because THREE.SkinnedMesh and THREE.InstancedMesh inherit from THREE.Mesh, the instanceof check evaluates to true. This means applyBVH will spend CPU time and memory generating a bounds tree on them. However, SkinnedMesh and InstancedMesh override .raycast() on their own prototypes in three.js. Since enableAcceleratedRaycast() only patches THREE.Mesh.prototype.raycast, the BVH acceleration will actually be ignored for them, falling back to stock raycasting anyway.

Recommendation: You may want to skip computing the bounds tree for them to save time and memory:

if (
  obj instanceof THREE.Mesh && 
  !(obj instanceof THREE.SkinnedMesh) && 
  !(obj instanceof THREE.InstancedMesh) &&
  obj.geometry
) {

thanks Ruofei! that makes sense as per the rationale

and great points, yeah so have updated the docstring so hopefully it makes it a bit easier to know when to apply, dense/static/environmental only, not UI/primitives/dynamic. portals only applies it on its own script subtree, not xb.core.scene, gallery part is the most important here cause the rings, etc.

skinned skip yep. on the instanced one though, i don't think gemini's right, InstancedMesh.raycast calls _mesh.raycast() per instance, which goes through the patched Mesh.prototype and short-circuits to geometry.boundsTree. so the bvh actually does get consulted on the shared geometry, kept it in. swapped in a BatchedMesh skip though, that one really does need the batched helpers (computeBatchedBoundsTree builds per-range trees on this.boundsTrees).

added a test that asserts the per-subclass policy, that will hopefully prevent drifts for this part

@ruofeidu
ruofeidu merged commit 333bf2f into google:main Jun 16, 2026
8 checks passed
ruofeidu pushed a commit that referenced this pull request Jun 17, 2026
… THREE inline

#372 landed the BVH helpers in the SDK (utils/BVHRaycast). drop the
direct three-mesh-bvh import + manual Mesh/BufferGeometry prototype
patch and call xb.enableAcceleratedRaycast() instead, which installs
the same computeBoundsTree / disposeBoundsTree helpers via the SDK's
dynamic import.

the per-detect computeBoundsTree() / disposeBoundsTree() calls stay,
they now resolve to the SDK-installed helpers. computeBoundsTree() is
guarded since enableAcceleratedRaycast() is async (dynamic import),
so an early Detect press before it resolves falls back to the stock
raycaster instead of throwing.

importmap keeps three-mesh-bvh (the SDK dynamic-imports it from the
page's module graph at runtime), bumped to 0.9.10 to match the
version the SDK is built against.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

algorithm spatial algorithm

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants