SkinnedMeshes aren't displayed when a parent is scaled to zero

[hybridherbst]

Describe the bug

Seems the calculations for culling away animated skinned meshes aren't fully correct; in this example file, the mesh disappears a lot randomly, especially in the later part of the file.

To Reproduce

Steps to reproduce the behavior:

In model-viewer

1. Go to https://modelviewer.dev/editor
2. Unpack + Drop SkinningCullingBug-truncated.zip
3. Select the 🖊️ pen Icon
4. Toggle "Play" off
5. Scrub to 250
6. Toggle "Play" back on
7. Watch the character pop in and out of view

In three.js/editor

1. Go to https://threejs.org/editor/
2. Unpack + Drop SkinningCullingBug-truncated.zip
3. Add a Directional Light
4. Select "SkinningCullingBug.glb", set animation time scale to 10 (the bug is more apparent in later parts of the animation when the character is scaled up)
5. Press Play
6. Watch the animation, you can set the timescale back to 1 (or lower) when the Character becomes bigger to see the issue better

Live example

Here's a video:
https://user-images.githubusercontent.com/2693840/167219369-588fe804-8fab-4c42-a5b4-79d4d38250dd.mp4

If you drop the same file into https://sandbox.babylonjs.com/, the character is visible all the time (as expected).

Expected behavior

Character is always visible

Platform:

• Device: [Desktop, Mobile]
• OS: [Windows]
• Browser: [Chrome]
• Three.js version: [latest on editor]

EDIT: replaced with a truncated version of the animation, original here: SkinningCullingBug.zip

[donmccurdy]

Unfortunately a long-standing issue: #14499

Current workarounds are to disable frustum culling for SkinnedMesh objects or to upscale the bounding box of the geometry by some factor.

[hybridherbst]

Oh no :(

I don't have control over threejs in most cases, e.g. when using model-viewer or other existing viewers.

For reference, @mrdoob had mentioned here

• Resting Bounds for skinned meshes are calculated without taking initial bind pose into account google/model-viewer#3189 (comment)

that he came up with an efficient solution; but I'm not sure that's already implemented anywhere?

[Mugen87]

Marking this as duplicate of #14499.

[mrdoob]

@hybridherbst Maybe you could make a PR in the modelviewer repo disabling frustum culling on SkinnedMesh?

[hybridherbst]

I actually looked into that but found that it's supposed to already be off!

• https://github.com/google/model-viewer/blob/f085bc1d2f46b851b95656713ec0b8ae9bbd9649/packages/model-viewer/src/three-components/gltf-instance/ModelViewerGLTFInstance.ts#L57

@elalish do you maybe have an idea what's going on? I can open an issue on model-viewer if you want of course. There's even a unit test that ensures frustum culling is off.

[hybridherbst]

Turns out turning off frustumCulled in the three.js/editor also doesn't affect culling. Maybe there's another issue here? I can confirm that in model-viewer frustumCulled is off for these meshes (actually for all) but the mesh is still culled (or, not rendered).

[hybridherbst]

@Mugen87 can I rename the issue to "SkinnedMeshes ignore the frustumCulled property" and could you reopen it?

[Mugen87]

Done!

[Mugen87]

Can you please set the frustumCulled property to false for the skinned mesh itself and for all its descendants (via traverse())?

If an object gets frustum culled, it does only affect the object itself but not its children.

[hybridherbst]

Can there be invisible descendants for a skinned mesh? In the above screenshot from the three.js editor that's the leaf node, there's no further children as far as I can see.

model-viewer does this:

scene.traverse((node: Object3D) => {
      ...
      // Three.js seems to cull some animated models incorrectly. Since we
      // expect to view our whole scene anyway, we turn off the frustum
      // culling optimization here.
      node.frustumCulled = false;
      ...
});      

[hybridherbst]

Updated the original post with a truncated animation that only shows the part with the biggest issues.

The issue seems to stem from the fact that the "LOD0" object (the parent of the SkinnedMeshRenderer) is scaled to 0 in some frames, and then the SkinnedMesh isn't visible - that might even be expected / by design - usually parent scale shouldn't affect how bones transform a mesh, but with scale=0 it's kind of undefined. Babylon/Unity seem to still show the mesh in that case, but three doesn't. Should it?

I'll look into it some more to verify that this is the issue.

[donmccurdy]

Just to clarify, are you describing one of these two scenarios? Or something else?

(A)

• Object3D
  • Bone1
  • Bone2
  • ...
• Object3D (scale=0)
  • SkinnedMesh

(B)

• Object3D (scale=0)
  • Bone1
  • Bone2
  • ...
• Object3D
  • SkinnedMesh

[hybridherbst]

Scenario A it is - usually the object scale doesn't affect the final result for skinned meshes, since the bones have the ultimate say, but for scale=0 I can see how that might be undefined.

For Scenario B, it would certainly be expected that the result isn't visible (but the skeleton helper / bones would also have scale 0 then).

[hybridherbst]

Here's a minimal repro model:
BentCylinder_Scales.zip

From left to right:

• SkinnedMesh is scaled to 1 (in this case, 100x smaller than the bind pose)
• SkinnedMesh is scaled to 100 (same as in bind pose)
• SkinnedMesh is scaled to 0.0001
• SkinnedMesh is scaled to 0
• SkinnedMesh has a position shift=10000

three.js/model-viewer:

Babylon:

Unity:

Gestaltor: (yet another variant! The tiny one disappears same as the 0-scale one)

And here's what happens if the values become more extreme, lighting breaks I guess:
BentCylinder_Scales_Extreme.zip

The black ones have the SkinnedMesh scaled to 1e11 and 1e-20, respectively.

[elalish]

This has the look of a numerical problem; maybe generating NaNs somewhere?

[hybridherbst]

Yep - I think the questions here would be "is this behaviour defined in glTF" and/or "should three have the same result as Babylon and Unity". Answer might be "this is too much of an edge case" :)

[donmccurdy]

glTF Validator should warn about these cases with NODE_SKINNED_MESH_NON_ROOT. While the glTF spec says the transform of a skinned mesh must be ignored, skinning is implemented very differently across engines (perhaps Unity especially...) and it's hard to guarantee that a non-invertible matrix — or reaching precision limits — is not going to have side effects.

If I remember correctly, Unity keeps a skinned mesh out of the scene hierarchy entirely, and the transform is completely ignored. three.js does at least compute transforms all the way down, a SkinnedMesh does not get special treatment there, and under some bind modes the SkinnedMesh world matrix can be optionally applied.

GLTFLoader does not change the default, bindMode = "attached". That's unintuitive to me, "detached" would sound like what the glTF specification calls for, but it has behaved correctly and ignored parent transforms (other than this scale=0 case) so far.

If there's a simple change in GLTFLoader (e.g. bindMode -> "detached") or WebGLRenderer (e.g. upload identity world matrix for a "detached" SkinnedMesh?) I think we could make that change, I don't think we could make more complex changes for this case.

/cc #17926 I thought there was code somewhere that simply prevented objects with zero scale from rendering, as a workaround for certain cases like flip-board visibility animation. But I can't find that code now, perhaps I was incorrect.

[hybridherbst]

Interesting, the spec even explicitly calls out that implementations MUST not hide a SkinnedMesh if all its axes are scaled to zero:
https://www.khronos.org/registry/glTF/specs/2.0/glTF-2.0.html#:~:text=When%20the%20scale,to%20zero%20simultaneously.

Implementation Note
When the scale is zero on all three axes (by node transform or by animated scale), implementations are free to optimize away rendering of the node’s mesh, and all of the node’s children’s meshes. This provides a mechanism to animate visibility. Skinned meshes must not use this optimization unless all of the joints in the skin are scaled to zero simultaneously.