-
-
Notifications
You must be signed in to change notification settings - Fork 35.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add support for skinning >4 bones per vertex with a bone weight texture #26222
base: dev
Are you sure you want to change the base?
Commits on Jun 8, 2023
-
Add support for skinning >4 bones per vertex with a bone weight texture
Related issue: mrdoob#26137 (mrdoob#26137) This PR adds an alternative skinning approach to the existing bone index/weight vertex buffers which uses a vertex shader texture to support an arbitrary amount of bone influences per vertex. The current vertex buffer skinning approach limits meshes to <= 4 bone influences per vertex. For example, a mesh with the following bone weights/indices in vertex buffers... ``` WEIGHTS_0 JOINTS_0 --------------- -------- [index] 0 1 2 3 0 1 2 3 v0: 0 1.0 0 0 0 1 2 3 v1: 0.1 0.7 0.1 0.1 0 1 2 3 ``` becomes this array/texture of (index, weight) pairs: ``` v0 v1 -------- ----------------------------------- [index] | 0 | 1 | 2 3 4 5 | 6 data: (1, 1.0) (-1, -1) (1, 0.7) (0, 0.1) (2, 0.1) (3, 0.1) (-1, -1) ⬆ ⬆ ⬆ sentinel value bone index weight ``` and this vertex buffer: ``` weights texture start index --------------------------- v0: 0 v1: 2 ``` *I have worked on this change as part of my job at Google since my org has projects which would like Three.js to have this feature.* **What changed:** 1. If a model has more than 1 bone weight buffer (>4 weights per vertex) then a bone weight texture is created and the shader behavior is changed to use it instead of vertex buffer skinning. 2. Vertex weight/index buffers are sorted by weight -- across all buffers -- before normalization in the buffers that will be used. - This allows skin texture creation to only include non-zero weights. - This "fixes" some artifacts for vertex buffer skinning when models have >4 weights and the higher weights are not in the first 4 weights. - This also means that loading skinned meshes takes longer because the per-vertex weights are sorted in addition to being normalized. **What did not change:** 1. The vertex buffer skinning approach is still the default for models that have at most 4 bone weights per vertex. **What needs decisions:** 1. Default behavior and how to control which skinning method to use - Currently defaults to the old buffer method if a model has <= 4 weights otherwise the texture method is used. - Loader/mesh options have controls for **always** or **never** using the texture approach. 3. Do we still want to keep the old method? - The code would be simpler without it and I haven't seen a big performance impact. - Main tradeoff seems to be model load time (creating the skinning texture from the vertex buffers) - I'm unsure if other parts of three.js or clients require/assume the presence of the skinning buffer **What is still being worked on** 1. Tests 2. Documentation 4. Adding support to loaders of file formats other than glTF. **Examples** 1. webgl_animation_skinning_many_bone_influences.html - Shows the difference between < 4 and >= 4 bone skinning for a model that was created with 16 bone skinning. - Notice: - The artifacts on the upper lip (left image, only 4 weights) - The difference in how much the nose gets pulled up/down with the mouth movement. ![frontal-close-up-4-vs-16-bones](https://github.com/mrdoob/three.js/assets/3453535/249a5233-97bc-4576-bdd9-fed9d071fb40) 5. webgl_animation_skinning_performance.html - loads many skinned meshes playing animations with a toggle between the old and new behavior to see if there are performance differences. ![perf-test](https://github.com/mrdoob/three.js/assets/3453535/7173d049-b846-4643-b297-47bbf1a1e9c3) **Performance** All performance numbers are from a 2019 MacBook Pro running Chrome 114.0.5735.106. **Framerate** `webgl_animation_skinning_performance.html` renders at the same 23 fps for both the vertex buffer and vertex texture skinning methods. The Chrome profiling image below shows that GPU code is executing for about 25% of the frame (11ms / 45ms) and vertex skinning is only a portion of that so this benchmark scene doesn't do the best job of stressing the vertex skinning: <img width="2231" alt="perf-test-profile" src="https://github.com/mrdoob/three.js/assets/3453535/fe629821-e8f8-42fb-adc8-7de5ab4ea00a"> That being said, the soldier model still seems like a realistic asset that someone would use which is why I used it in the benchmarking scene. If there are other animated models with higher vertex counts that would increase GPU vertex shader runtime differences, I am happy to try them. **Note:** Texture skinning could have beneficial performance for models with <= 4 weights per vertex because it strips out weights that are zero. For the `Soldier.glb` model, this removed 38% of the weights. **Model Loading** method | `Soldier.glb` (7434 vertices, 4 weights) runtime| `HeadWithMax16Joints.glb` (2474 vertices, 16 weights) runtime --------|---------------------------|--------------- `SkinnedMesh.normalizeSkinWeights (dev)` | 5 ms | N/A `SkinnedMesh.normalizeSkinWeights (this PR)` | 12 ms | 21 ms `SkinnedMesh.createBoneIndexWeightsTexture` | 7 ms | 8 ms `GLTFLoader.load` (buffer skinning) | 190-380 ms | N/A `GLTFLoader.load` (texture skinning) | 190-380 ms | 240-290 ms The new implementation of `SkinnedMesh.normalizeSkinWeights()` is slower because it sorts weights in addition to normalizing them. It could be made faster by sorting each vertex's buffer data in-place instead of copying to separate arrays and then copying them back. The total load time and variance of the load time was so large that the additional processing in `normalizeSkinWeights` and `createBoneIndexWeightsTexture` did not have a noticeable effect.
Configuration menu - View commit details
-
Copy full SHA for 34d7408 - Browse repository at this point
Copy the full SHA 34d7408View commit details
Commits on Jun 9, 2023
-
Configuration menu - View commit details
-
Copy full SHA for fc03f04 - Browse repository at this point
Copy the full SHA fc03f04View commit details -
Add screenshots for 2 new skinning examples
This also adds them to examples/files.json and updates one of the titles The screenshots were generated with the following command after fine-tuning them to produce a good screenshot: ``` npm run make-screenshot \ webgl_animation_skinning_many_bone_influences \ webgl_animation_skinning_performance ```
Configuration menu - View commit details
-
Copy full SHA for fb59366 - Browse repository at this point
Copy the full SHA fb59366View commit details -
Configuration menu - View commit details
-
Copy full SHA for 94b7331 - Browse repository at this point
Copy the full SHA 94b7331View commit details -
Increase e2e-test render timeout and parse time to avoid flaky tests
One of the MacOS e2e tests was failing on the new webgl_animation_performance example page which might be due to rendering time. I'm making this change to see if it fixes things in CI. The E2E test works fine on my 2019 MacBook pro.
Configuration menu - View commit details
-
Copy full SHA for 9183fd0 - Browse repository at this point
Copy the full SHA 9183fd0View commit details -
Configuration menu - View commit details
-
Copy full SHA for b000985 - Browse repository at this point
Copy the full SHA b000985View commit details
Commits on Jun 13, 2023
-
Add env map and tone mapping for new >4 bone skinning examples
Directional light was also removed from the "many bones" example because it wasn't needed. The env map would make more of a difference if the material had lower roughness but it looks too bright when extra lights are added like in the "performance" example even though tone mapping is on.
Configuration menu - View commit details
-
Copy full SHA for 461adde - Browse repository at this point
Copy the full SHA 461addeView commit details -
Configuration menu - View commit details
-
Copy full SHA for 6aab6e7 - Browse repository at this point
Copy the full SHA 6aab6e7View commit details -
Add bone weight texture support to all SkinnedMesh loaders
This is done by moving the weight buffer normalization and texture creation into the SkinnedMesh constructor. All relevant loaders now have an option that is given to the SkinnedMesh constructor which controls whether or not a weight texture is created.
Configuration menu - View commit details
-
Copy full SHA for 0de2851 - Browse repository at this point
Copy the full SHA 0de2851View commit details -
Configuration menu - View commit details
-
Copy full SHA for d9c777d - Browse repository at this point
Copy the full SHA d9c777dView commit details -
Add directional light back into >4 bone skinning example for better v…
…isuals The visual artifacts are easier to see when there's a directional light in the scene.
Configuration menu - View commit details
-
Copy full SHA for 4aebe19 - Browse repository at this point
Copy the full SHA 4aebe19View commit details -
Configuration menu - View commit details
-
Copy full SHA for 6d8e7f1 - Browse repository at this point
Copy the full SHA 6d8e7f1View commit details
Commits on Jun 14, 2023
-
Configuration menu - View commit details
-
Copy full SHA for 6b15533 - Browse repository at this point
Copy the full SHA 6b15533View commit details -
Configuration menu - View commit details
-
Copy full SHA for d47fa40 - Browse repository at this point
Copy the full SHA d47fa40View commit details -
Configuration menu - View commit details
-
Copy full SHA for aa98ea6 - Browse repository at this point
Copy the full SHA aa98ea6View commit details
Commits on Jun 17, 2023
-
Configuration menu - View commit details
-
Copy full SHA for 86d53a3 - Browse repository at this point
Copy the full SHA 86d53a3View commit details -
Configuration menu - View commit details
-
Copy full SHA for 5d46ce1 - Browse repository at this point
Copy the full SHA 5d46ce1View commit details
Commits on Jun 20, 2023
-
Remove flaky webgl_animation_skinning_performance from screenshot tests
This new example takes a long time to load and can have a low framerate so the exact state of the animations when the screenshot is taken will be slightly different each time. This causes the screenshot test to be flaky.
Configuration menu - View commit details
-
Copy full SHA for f4cfced - Browse repository at this point
Copy the full SHA f4cfcedView commit details -
Configuration menu - View commit details
-
Copy full SHA for aa34feb - Browse repository at this point
Copy the full SHA aa34febView commit details
Commits on Jul 26, 2023
-
Configuration menu - View commit details
-
Copy full SHA for 7721604 - Browse repository at this point
Copy the full SHA 7721604View commit details -
Restore old visuals after useLegacyLights default changed in dev
See: 1. mrdoob#26392 2. mrdoob#26267 The default behavior of light intensities was changed. Some examples had the light intensity values changed from 1 to 3 to preserve the old behavior. This is necessary after merging dev into this branch. The screenshots still visually look the same but I am committing them as well in case there were any subtle changes due to this change in lighting behavior.
Configuration menu - View commit details
-
Copy full SHA for 4ff7701 - Browse repository at this point
Copy the full SHA 4ff7701View commit details -
Update svg_sandbox screenshot because e2e test is flaky
I did not change any behavior related to this screenshot test. One E2E test run said it differed in 2.5% of pixels but the other test runs were fine. I'm updating the screenshot to hopefully make this test less flaky.
Configuration menu - View commit details
-
Copy full SHA for 50a6da9 - Browse repository at this point
Copy the full SHA 50a6da9View commit details
Commits on Aug 7, 2023
-
Configuration menu - View commit details
-
Copy full SHA for d66eab4 - Browse repository at this point
Copy the full SHA d66eab4View commit details
Commits on Sep 11, 2023
-
Configuration menu - View commit details
-
Copy full SHA for b5e206e - Browse repository at this point
Copy the full SHA b5e206eView commit details
Commits on Oct 17, 2023
-
Configuration menu - View commit details
-
Copy full SHA for be0b73d - Browse repository at this point
Copy the full SHA be0b73dView commit details
Commits on Nov 9, 2023
-
Configuration menu - View commit details
-
Copy full SHA for fb8a84e - Browse repository at this point
Copy the full SHA fb8a84eView commit details
Commits on Jan 22, 2024
-
Configuration menu - View commit details
-
Copy full SHA for 8ca2e48 - Browse repository at this point
Copy the full SHA 8ca2e48View commit details
Commits on Apr 3, 2024
-
Configuration menu - View commit details
-
Copy full SHA for d3aa1b4 - Browse repository at this point
Copy the full SHA d3aa1b4View commit details -
I accidentally added spaces instead of tabs during a previous merge commit that required manual resolving.
Configuration menu - View commit details
-
Copy full SHA for 607c88a - Browse repository at this point
Copy the full SHA 607c88aView commit details