perf(world/faces): BVH-accelerate per-landmark depth raycasts - #367
Merged
Conversation
This was referenced Jun 15, 2026
salmanmkc
force-pushed
the
feat/face-bvh-raycast
branch
from
June 15, 2026 04:44
70620bc to
80802af
Compare
salmanmkc
changed the title
Collaborator
|
I don't think we want |
salmanmkc
added a commit
to salmanmkc/xrblocks
that referenced
this pull request
Jun 15, 2026
per review feedback on google#367, three-mesh-bvh should be optional like troika-three-text is for TextView: type-only import at build time, dynamic import at runtime with try / catch + status tracking. apps without three-mesh-bvh installed (or without it in their importmap) keep working with the stock three.js raycaster. changes: - moved three-mesh-bvh from dependencies to devDependencies (still needed for the SDK's own build + tests, but not forced on consumers) - import type * as BVH from 'three-mesh-bvh' for compile-time types - enableAcceleratedRaycast() and applyBVH() are now async; they kick off the dynamic import on first call and share the same promise on subsequent calls - on import failure, both helpers log a one-line warn and return false (or no-op for applyBVH). raycasts fall back to the stock walker - new isBVHReady() sync check for consumers that want to know whether the patches are installed - disposeBVH() stays sync; no-op when BVH was never loaded tests updated to await the async helpers. 295 tests still passing. mirrors the dynamic-import pattern in src/ui/components/TextView.ts for troika-three-text.
dli7319
approved these changes
Jun 15, 2026
dli7319
force-pushed
the
feat/face-bvh-raycast
branch
from
June 15, 2026 17:25
7466be5 to
774a9f1
Compare
…casts
The SDK's per-frame interaction raycaster walks every triangle of
every mesh under xb.core.scene via THREE.Raycaster.intersectObject
(recursive). For demos with non-trivial scene geometry (e.g. portals
with 5 loaded immersive worlds), the per-triangle walk dominates the
main thread.
three-mesh-bvh's accelerated raycaster solves this but it has to be
opted into per geometry. Adds two public helpers:
- enableAcceleratedRaycast(): installs the THREE.Mesh prototype patch
and BufferGeometry helpers. Idempotent across callers (so multiple
subsystems can ping it).
- applyBVH(root, {recursive?}): walks the tree and builds a bounds
tree on every mesh's geometry. Demos opt in by calling this on the
root of their scene after meshes are added.
Geometries without a bounds tree continue to use the stock raycaster,
so flipping the prototype patch on globally is safe for existing code
that never calls applyBVH.
Also adds three-mesh-bvh to package.json + rollup externals so it
ships consistently across the SDK and addons. 6 new tests cover
prototype patch install, idempotency, recursive build, no-rebuild,
recursive=false stop, and dispose.
Portals demo loads 5 immersive scenes into xb.core.scene. The SDK's per-frame interaction raycast (Input.performRaycastOnScene) walks every triangle under the scene root each controller frame, and the per-triangle walk dominated the main thread in a perf trace before this. Calls xb.applyBVH(this) at the end of PortalGalleryScene.init() so the bounds tree is built once per geometry after all immersives are added. Subsequent raycasts go through three-mesh-bvh's accelerated path (O(log triangles) per ray instead of O(triangles)). Adds three-mesh-bvh to the demo importmap. Same version pinned by the SDK and other demos.
FaceLandmarker emits 478 landmarks per face and processFaceLandmarkerResult raycasts each one against the depth-mesh snapshot to project to world coords. Stock three.js raycaster is O(triangles) per ray; depth mesh runs in the thousands of triangles, so the per-detection raycast loop alone can dominate the frame budget even with inference moved to a worker. Wires three-mesh-bvh's prototype patches at FaceRecognizer module load (mirrors what objects_3d demo already does) and calls computeBoundsTree() on the cloned depth-mesh geometry once per detection in getDepthMeshSnapshot. Meshes without a BVH fall back to the stock walker so the patch is safe to apply globally. The dep is already in the addons matrix; this just adds it to the SDK package.json + the face_mirror demo importmap. Same version objects_3d uses.
BVH-building over the cloned depth-mesh geometry was running every detection (computeBoundsTree on a few-thousand-triangle mesh, ~30 ms). For a static desktop sim the depth source rarely changes between detections so the same BVH is valid for many in a row. getDepthMeshSnapshot now caches the cloned snapshot and its BVH and reuses them while the source geometry's position attribute version is unchanged. World transform (position / quaternion / scale) still refreshes each call so the cached snapshot reflects whatever pose the depth mesh is at right now. When the source version bumps (three.js does this on needsUpdate = true), we dispose the previous BVH and re-clone. Synergistic with the inflight-only depth throttle (google#370): static scene = depth refreshes seldom = BVH built once and held.
Drop the inline three-mesh-bvh prototype patches and import enableAcceleratedRaycast() from src/utils/BVHRaycast (added in google#372). The helper is idempotent across modules so other subsystems that also install BVH (objects_3d, portals) share the same patch without fighting over the prototype. This PR's branch is stacked on google#372 so the util is available; merge google#372 first.
per review feedback on google#367, three-mesh-bvh should be optional like troika-three-text is for TextView: type-only import at build time, dynamic import at runtime with try / catch + status tracking. apps without three-mesh-bvh installed (or without it in their importmap) keep working with the stock three.js raycaster. changes: - moved three-mesh-bvh from dependencies to devDependencies (still needed for the SDK's own build + tests, but not forced on consumers) - import type * as BVH from 'three-mesh-bvh' for compile-time types - enableAcceleratedRaycast() and applyBVH() are now async; they kick off the dynamic import on first call and share the same promise on subsequent calls - on import failure, both helpers log a one-line warn and return false (or no-op for applyBVH). raycasts fall back to the stock walker - new isBVHReady() sync check for consumers that want to know whether the patches are installed - disposeBVH() stays sync; no-op when BVH was never loaded tests updated to await the async helpers. 295 tests still passing. mirrors the dynamic-import pattern in src/ui/components/TextView.ts for troika-three-text.
Now that google#372's enableAcceleratedRaycast is async and three-mesh-bvh is an optional dynamic import, the per-detection computeBoundsTree call in getDepthMeshSnapshot needs to gate on whether the BVH module has actually loaded. When BVH isn't available (no install, no importmap), we skip the bounds-tree build and the stock raycaster handles the per-landmark intersections. Functionally a no-op for any consumer that already has three-mesh-bvh in their importmap (face_mirror demo).
dli7319
force-pushed
the
feat/face-bvh-raycast
branch
from
June 15, 2026 17:25
774a9f1 to
c53b8b7
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
even with FaceLandmarker inference moved to a worker (#366), the per-detection raycast loop still tanked desktop FPS. perf trace showed
getVertexPosition+intersectTriangle+_computeIntersectionseating ~9 of every 14 main-thread seconds.processFaceLandmarkerResultraycasts each of the 478 landmarks against the depth-mesh snapshot to project to world coords. stock three.js raycaster is O(triangles) per ray, depth mesh runs in the thousands of triangles, so the per-detection raycast loop alone dominates the frame budget.wires three-mesh-bvh prototype patches at FaceRecognizer module load (same trick objects_3d uses) and calls
computeBoundsTree()on the cloned depth-mesh geometry ingetDepthMeshSnapshot. meshes without a BVH fall back to the stock walker so the patch is safe to apply globally.measured improvement in the same trace before/after:
getVertexPosition4284ms -> 88ms,intersectTriangle3211ms -> 57ms (~98% reduction on each).follow-up commit caches the cloned snapshot + BVH and reuses them while the source geometry's position attribute version is unchanged (three.js bumps it on
needsUpdate = true). world transform still refreshes each call so the cached snapshot reflects the current depth mesh pose. for a static desktop sim this amortizes the BVH build across all detections instead of running every detection.three-mesh-bvh added to package.json + face_mirror demo importmap. same version objects_3d already pins.
related: #366 (worker), #370 (simulator depth inflight guard). all independent file-wise.