Added raycasting support to THREE.SkinnedMesh.

[cpollard1001]

Ray-casting currently only supports deformations, not bone transformations. The bone transformations need to be applied after the morph changes so I created a function that takes the morphed vertices and their index for the skin parameters and returns the bone transformed vertex. The code was adapted from @makc at https://github.com/makc/three.js.fork/tree/skin-raycast. This version allows the morphs to be applied first and doesn't require a creator to be called.

[makc]

you mean skin can have both morph targets and bones??

[mrdoob]

Do you mind adding a example too?

[cpollard1001]

@mrdoob Sure, I'd be happy to make an example. I could use the skinned cyclinder in the docs, sway it back and forth, and have it change color on click for both BufferGeometry and Geometry. Should I add a new example.html file and commit it or would you like it in another format?

@makc Yes, morph targets are a property of THREE.Mesh. In the GPU shader, the morph targets are applied first, say a person's hand closing, then those vertices are deformed according to the bone structure. If bones are applied first, the relative offsets won't be applied in the correct directions.

[cpollard1001]

I added the example and the calculations seem to be working well. One big gotcha is that you have to set the geometry.boundingSphere correctly to detect collisions outside the geometry's bounding sphere. There are several different methods for improving the performance. We could cache vertices by index so that they aren't recalculated multiple times for different faces. We could also calculate them all up front and use that to create the bounding box. This would add a lot of computation so maybe we could add a flag to enable/disable this. Or we could just leave it as it is.

Demo link: http://ci.threejs.org/api/pullrequests/8953/examples/webgl_raycast_skinning.html

[WestLangley]

/bump

[WestLangley]

/bump again

/ping @Mugen87 Do we want to support this?

And do we want to support raycasting against BufferGeometry with morph targets?

[donmccurdy]

SkinnedMesh no longer supports THREE.Geometry, so this can be removed.

[donmccurdy]

This clone variable is shadowed (re-declared) in the function below, and isn't getting reused. Ideally, I think I would overwrite the vertex parameter passed by the user rather than overwriting a single vector on subsequent calls.

[Mugen87]

Do we want to support this?

TBH, I have concerns about this, see #6911 (comment)

I think there are enough users outside there that will use SkinnedMesh.raycast() per frame and then complain about bad performance.

In many games, you normally don't raycast against the animated (maybe complex) geometry which is intended for rendering. You have some sort of bounding volume hierarchy or maybe just a single tight bounding volume which makes raycasting more efficient. Sometimes it's also possible to use totally different approaches like GPU picking.

I think it's okay to add it but it should be well documented.

[donmccurdy]

Aside from the lack of a bounding volume in this PR, I don't see why raycasting against a SkinnedMesh should be dramatically slower. Still O(n) in the number of faces, yes?

I actually think the bounding volumes of the geometry should be used here, as they are in THREE.Mesh. If the user wants to expand those volumes (say, 20x) they can do so manually. threejs should probably not be responsible for knowing how much of a buffer to put on the bounding volumes, though. We'd need to document that.

More importantly, we should just profile this. Maybe on a simple model (RobotExpressive.glb) and a more complex one (Samba Dancing.fbx).

Another strategy is to put bounding volumes on the bones, and raycast against those. But I think this would require some user input.

[Mugen87]

More importantly, we should just profile this. Maybe on a simple model (RobotExpressive.glb) and a more complex one (Samba Dancing.fbx).

Definitely 👍 . I have no performance numbers but applying the vertex transformation related to skinning each frame can be very expensive.

The code from this fiddle might be a good starting point: https://jsfiddle.net/fnjkeg9x/1/

It computes the AABB for an SkinnedMesh per frame. In runs fine on my iMac but slow on my Pixel (1).

[metipton]

I was playing around with this and instead of expanding the aabb for each vertex, I just expanded for every 128th vertex. It's significantly faster and I saw no difference in the accuracy of the bounding box. You might be able to add a sampling parameter that could allow for skipping indices if speed is desired over accuracy

[titansoftime]

Another strategy is to put bounding volumes on the bones, and raycast against those. But I think this would require some user input.

YES PLEASE!!!

[yu-takada]

I think putting bounding volumes on the bones is a very useful too, if there are no affection on other features.

[gkjohnson]

In my opinion this should be supported especially with morph target raycasting now being available (#15985). While raycasting a SkinnedMesh may be slow in some complex cases it's hard for me to see that the current behavior (returning a visually incoherent intersection point) would be preferred. In simple cases transforming every face before raycasting is a fine solution and I think it's understood that as a users application outgrows the simple solutions that custom, application-specific solutions will need to be used.

You have some sort of bounding volume hierarchy or maybe just a single tight bounding volume which makes raycasting more efficient. Sometimes it's also possible to use totally different approaches like GPU picking.

Using a lower LoD skinned mesh for raycasting when the rendered mesh is sufficiently complex is another solution. Regarding putting bounding volumes on bones this would need to be tunable per-model / bone to fit the skinned mesh volume, right?

I'm not sure what the right solution is to the bounding volume problem, though. This remains a problem for frustum culling SkinnedMeshes, right? Is there a solution to this that can be used to solve both issues? Or should it be up to the application to manually specify bounds that encompass the extents that an animated mesh is expected to reach?

[Mugen87]

'm not sure what the right solution is to the bounding volume problem, though.

From my point of view, this is the main challenge. Not the actually raycasting logic. Computing the current vertex displacement is easy to implement. However, if wrong bounding volumes are used, raycasting will fail no matter how efficient its implementation is. The AABB is an important part of SkinnedMesh.raycast() since the method should use bounding volumes for early out testing. Not doing this will make this method impractical for more complex scenes.

Computing the max AABB for a geometry with morph targets was easy since all data you need for this are available in the geometry (morphAttributes). This is not true for skinning. Hence, the equivalent AABB for a geometry of a skinned mesh is more complicated since you can't solve this task on geometry level. You need the skeleton and the keyframe data so you know how the bones are animated over time and how they influence the geometry.

The problem is that in three.js animation clips are decoupled and manage "somewhere" in the user code. That makes it hard to automate the computation of a proper AABB since the library does not have a defined place for accessing the animations of a skinned mesh.

You essentially need similar logic like from this fiddle that I've written some time ago for a forum post:

https://jsfiddle.net/v02kLhsm/2/

The red box represents the current AABB, the green the better one which encloses the maximum extension of the geometry (similar to morph targets). As you can see, even the computation of this AABB is a pain since you have to sample the clip over the entire duration and expand the AABB step by step. Unfortunately, the result is just an approximation. The real AABB could be bigger. The problem is that the used sampling is much easier that processing keyframe by keyframe since position and quaternion tracks might have different time values and keyframe counts (which means you would have to interpolate missing entries. Ugh!).

Ideally, SkinnedMesh could automatically compute such an AABB. However, since it has no access to animation data, a different approach is needed. The only solution I can think of is introducing SkinnedMesh.computeBoundingVolumes() and pass in an array of animation clips. Users have to call this method after creating an instance of SkinnedMesh or when a new geometry is assigned. A proper AABB would solve the broken view frustum culling support for skinned meshes and enable more performant raycasting.

[donmccurdy]

@Mugen87 if I understand correctly, you're proposing that the SkinnedMesh AABB would represent bounds within which the skinned mesh is permanently contained, when any of the provided animation clips are played? In other words, the union of bounds from each pose in each frame of each clip?

That may well be the right way to go — it has the advantage of being precompute-friendly — but I'll point out one limitation. It doesn't support procedural animation, IK, or ragdoll physics.

I'd suggested above that bounding volumes on the bones would require user input, but I'm not sure that's true. Suppose we:

1. Process the mesh to compute a volume for each bone, defined by the positions of all vertices weighted to that bone.
2. Store the bone volumes somewhere on the SkinnedMesh or Skeleton (since the Bones could be used by multiple meshes). For example, mesh.skeleton.boneVolumes.
3. When SkinnedMesh.raycast is called, it will update the bone volume positions to the positions of each bone at that frame, then:
   a. efficiently compute the AABB for the entire mesh from the bone volumes
   b. or, just raycast directly against the bone volumes

I believe that should be fairly efficient, although raycasting a SkinnedMesh with 200+ bones would still require testing 200 AABBs. That can also be solved, but maybe let's consider that later...

[Mugen87]

if I understand correctly, you're proposing that the SkinnedMesh AABB would represent bounds within which the skinned mesh is permanently contained, when any of the provided animation clips are played? In other words, the union of bounds from each pose in each frame of each clip?

Yes, it's the equivalent to Unity's SkinnedMeshRenderer.localBounds:

https://docs.unity3d.com/ScriptReference/SkinnedMeshRenderer-localBounds.html

Having this AABB will solve the view frustum culling issue and raycasting for most use cases. Yes, the approach has limitations as you pointed out. But I think I wouldn't go down the path of AABBs per bone now. Even if you have these, you would still need a single maximum AABB. So we can start with this one and refine the approach later.

[donmccurdy]

I might suggest starting with an AABB derived from the default pose alone (whereas the current implementation ignores this pose), so the user is not required to have a list of clips to get an accurate AABB. That would fix cases where an FBX file (or a file converted from FBX to something else) has a skeleton with a 100x transform and the AABB is 100x too small.

Expanding that bounding box further with animation clips could be a future option, and/or the bone volume approach.

[gkjohnson]

Process the mesh to compute a volume for each bone, defined by the positions of all vertices weighted to that bone.

Oh you're right I hadn't understood that -- I really like that idea.

I believe that should be fairly efficient, although raycasting a SkinnedMesh with 200+ bones would still require testing 200 AABBs. That can also be solved, but maybe let's consider that later...

I think there are cases all over the library where extreme scenarios lead to poor performance. I think it's understood (or should be) that three.js provides correct behavior at reasonable performance in common use cases and that more advanced cases will require custom solutions. That's at least how I've thought about things so I'm not sure if I see the 200+ bone case as a problem.

Even if you have these, you would still need a single maximum AABB.

While it might not be a tight containment an AABB could be computed from the bone AABBs more quickly which means it could be computed more effectively every frame like in your example (or from a set of sampled animation frames). Implementation aside why is a single maximum AABB needed? Where would it go? It doesn't seem like it belongs on the BufferGeometry instance like we expect for Mesh because the AABB for a skinned geometry is a function of the skeleton and animation applied to it which may be different depending on which SkinnedMesh is using the geometry. It seems more like there should be a special case in Frustum.intersectsObject to handle SkinnedMesh checking that can take this into account.

[makc]

and why do bones boxes when you could just do some octree? it feels like there should be no improvement.

[donmccurdy]

@makc I'm not sure what you're suggesting, sorry. Even with an octree or other spatial data structure, it would be unreasonable to update the positions of each triangle in the octree on every frame, and colliders on bones would be more efficient.

[makc]

@donmccurdy so you mean fixed boxes around bones then? and just ignore if the triangle leaves the box volume?

[donmccurdy]

Fixed boxes, yes, but defined to contain the base position of every vertex the bone can influence. It's not guaranteed that vertices will stay within that box, but aside from some pretty contrived examples it should be quite accurate. And an object-level AABB derived from the bone volumes should be guaranteed to contain all vertices. If a user wants to override the autogenerated bone volumes, that's an option.

See also "bone colliders" in Unity... those volumes could be used for physics, not just raycasting.

[tsuchan]

Hi, just checking back about this issue, whether it's still in-progress... I'm wondering what's the current status. Thanks for your help!

[makc]

@tsuchan my guess is noone is doing anything :D well maybe @donmccurdy works on his bone colliders idea when he has nothing better to do

[donmccurdy]

Not any time soon for me, I have more draft PRs open than I'm keeping up with. 😅

I do support applying the bone transforms to vertices when raycasting, even if the bounding box calculation (for now) remains exactly as it is, disregarding bone transforms. This PR satisfies that, and seems like a good incremental step toward the more advanced solutions. I'll defer to @Mugen87 on how the code should be structured.

[tsuchan]

@donmccurdy I've tested your PR and it works fine for me. It is a big progress! I can use this in my project. It seems it just needs small changes to make it work with r113? Looking forward to see it implemented it in the next release.

[donmccurdy]

We shouldn't clone() in this method, it is likely to be called often. I would suggest modifying the input vertex instead, and perhaps changing the signature:

transformVertex ( index : number, target : THREE.Vector3 ) : this 

[donmccurdy]

Oh it's @cpollard1001's PR and not mine. :)

I think it needs a few fixes, and perhaps a different method name, but could perhaps be in r115 or r116?

[donmccurdy]

Had a go at fixing the feedback on this PR in #19178. I'll leave the bone collider idea for someone else. 😇

[Mugen87]

Closing in favor of #19178.