Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for skinning >4 bones per vertex with a bone weight texture #26222

Open
wants to merge 29 commits into
base: dev
Choose a base branch
from

Commits on Jun 8, 2023

  1. Add support for skinning >4 bones per vertex with a bone weight texture

    Related issue: mrdoob#26137 (mrdoob#26137)
    
    This PR adds an alternative skinning approach to the existing bone index/weight vertex buffers which uses a vertex shader texture to support an arbitrary amount of bone influences per vertex. The current vertex buffer skinning approach limits meshes to <= 4 bone influences per vertex.
    
    For example, a mesh with the following bone weights/indices in vertex buffers...
    
    ```
            WEIGHTS_0        JOINTS_0
            ---------------  --------
    [index] 0   1   2   3    0 1 2 3
        v0: 0   1.0 0   0    0 1 2 3
        v1: 0.1 0.7 0.1 0.1  0 1 2 3
    ```
    
    becomes this array/texture of (index, weight) pairs:
    
    ```
            v0                v1
            --------          -----------------------------------
    [index] |   0  |     1    |  2        3        4        5   |    6
      data: (1, 1.0) (-1, -1) (1, 0.7) (0, 0.1) (2, 0.1) (3, 0.1) (-1, -1)
                       ⬆                        ⬆   ⬆
                  sentinel value         bone index  weight
    ```
    
    and this vertex buffer:
    
    ```
        weights texture start index
        ---------------------------
    v0: 0
    v1: 2
    ```
    
    *I have worked on this change as part of my job at Google since my org has projects which would like Three.js to have this feature.*
    
    **What changed:**
    
    1. If a model has more than 1 bone weight buffer (>4 weights per vertex) then a bone weight texture is created and the shader behavior is changed to use it instead of vertex buffer skinning.
    2. Vertex weight/index buffers are sorted by weight -- across all buffers -- before normalization in the buffers that will be used.
        - This allows skin texture creation to only include non-zero weights.
        - This "fixes" some artifacts for vertex buffer skinning when models have >4 weights and the higher weights are not in the first 4 weights.
        - This also means that loading skinned meshes takes longer because the per-vertex weights are sorted in addition to being normalized.
    
    **What did not change:**
    
    1. The vertex buffer skinning approach is still the default for models that have at most 4 bone weights per vertex.
    
    **What needs decisions:**
    
    1. Default behavior and how to control which skinning method to use
        - Currently defaults to the old buffer method if a model has <= 4 weights otherwise the texture method is used.
        - Loader/mesh options have controls for **always** or **never** using the texture approach.
    3. Do we still want to keep the old method?
        - The code would be simpler without it and I haven't seen a big performance impact.
        - Main tradeoff seems to be model load time (creating the skinning texture from the vertex buffers)
        - I'm unsure if other parts of three.js or clients require/assume the presence of the skinning buffer
    
    **What is still being worked on**
    
    1. Tests
    2. Documentation
    4. Adding support to loaders of file formats other than glTF.
    
    **Examples**
    
    1. webgl_animation_skinning_many_bone_influences.html - Shows the difference between < 4 and >= 4 bone skinning for a model that was created with 16 bone skinning.
        - Notice:
            - The artifacts on the upper lip (left image, only 4 weights)
            - The difference in how much the nose gets pulled up/down with the mouth movement.
    
    ![frontal-close-up-4-vs-16-bones](https://github.com/mrdoob/three.js/assets/3453535/249a5233-97bc-4576-bdd9-fed9d071fb40)
    
    5. webgl_animation_skinning_performance.html - loads many skinned meshes playing animations with a toggle between the old and new behavior to see if there are performance differences.
    
    ![perf-test](https://github.com/mrdoob/three.js/assets/3453535/7173d049-b846-4643-b297-47bbf1a1e9c3)
    
    **Performance**
    
    All performance numbers are from a 2019 MacBook Pro running Chrome 114.0.5735.106.
    
    **Framerate**
    
    `webgl_animation_skinning_performance.html` renders at the same 23 fps for both the vertex buffer and vertex texture skinning methods. The Chrome profiling image below shows that GPU code is executing for about 25% of the frame (11ms / 45ms) and vertex skinning is only a portion of that so this benchmark scene doesn't do the best job of stressing the vertex skinning:
    
    <img width="2231" alt="perf-test-profile" src="https://github.com/mrdoob/three.js/assets/3453535/fe629821-e8f8-42fb-adc8-7de5ab4ea00a">
    
    That being said, the soldier model still seems like a realistic asset that someone would use which is why I used it in the benchmarking scene. If there are other animated models with higher vertex counts that would increase GPU vertex shader runtime differences, I am happy to try them.
    
    **Note:** Texture skinning could have beneficial performance for models with <= 4 weights per vertex because it strips out weights that are zero. For the `Soldier.glb` model, this removed 38% of the weights.
    
    **Model Loading**
    
    method | `Soldier.glb` (7434 vertices, 4 weights) runtime| `HeadWithMax16Joints.glb` (2474 vertices, 16 weights) runtime
    --------|---------------------------|---------------
    `SkinnedMesh.normalizeSkinWeights (dev)` | 5 ms | N/A
    `SkinnedMesh.normalizeSkinWeights (this PR)` | 12 ms  | 21 ms
    `SkinnedMesh.createBoneIndexWeightsTexture` | 7 ms | 8 ms
    `GLTFLoader.load` (buffer skinning) | 190-380 ms |  N/A
    `GLTFLoader.load` (texture skinning) | 190-380 ms | 240-290 ms
    
    The new implementation of `SkinnedMesh.normalizeSkinWeights()` is slower because it sorts weights in addition to normalizing them. It could be made faster by sorting each vertex's buffer data in-place instead of copying to separate arrays and then copying them back.
    
    The total load time and variance of the load time was so large that the additional processing in `normalizeSkinWeights` and `createBoneIndexWeightsTexture` did not have a noticeable effect.
    cstegel committed Jun 8, 2023
    Configuration menu
    Copy the full SHA
    34d7408 View commit details
    Browse the repository at this point in the history

Commits on Jun 9, 2023

  1. Configuration menu
    Copy the full SHA
    fc03f04 View commit details
    Browse the repository at this point in the history
  2. Add screenshots for 2 new skinning examples

    This also adds them to examples/files.json and updates one of the titles
    
    The screenshots were generated with the following command after fine-tuning
    them to produce a good screenshot:
    
    ```
    npm run make-screenshot \
    	webgl_animation_skinning_many_bone_influences \
    	webgl_animation_skinning_performance
    ```
    cstegel committed Jun 9, 2023
    Configuration menu
    Copy the full SHA
    fb59366 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    94b7331 View commit details
    Browse the repository at this point in the history
  4. Increase e2e-test render timeout and parse time to avoid flaky tests

    One of the MacOS e2e tests was failing on the new
    webgl_animation_performance example page which might be due to rendering time.
    
    I'm making this change to see if it fixes things in CI. The E2E test works fine
    on my 2019 MacBook pro.
    cstegel committed Jun 9, 2023
    Configuration menu
    Copy the full SHA
    9183fd0 View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    b000985 View commit details
    Browse the repository at this point in the history

Commits on Jun 13, 2023

  1. Add env map and tone mapping for new >4 bone skinning examples

    Directional light was also removed from the "many bones" example
    because it wasn't needed.
    
    The env map would make more of a difference if the material had lower roughness
    but it looks too bright when extra lights are added like in the "performance"
    example even though tone mapping is on.
    cstegel committed Jun 13, 2023
    Configuration menu
    Copy the full SHA
    461adde View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    6aab6e7 View commit details
    Browse the repository at this point in the history
  3. Add bone weight texture support to all SkinnedMesh loaders

    This is done by moving the weight buffer normalization and texture creation
    into the SkinnedMesh constructor. All relevant loaders now have an option
    that is given to the SkinnedMesh constructor which controls whether or not a
    weight texture is created.
    cstegel committed Jun 13, 2023
    Configuration menu
    Copy the full SHA
    0de2851 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    d9c777d View commit details
    Browse the repository at this point in the history
  5. Add directional light back into >4 bone skinning example for better v…

    …isuals
    
    The visual artifacts are easier to see when there's a directional light in
    the scene.
    cstegel committed Jun 13, 2023
    Configuration menu
    Copy the full SHA
    4aebe19 View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    6d8e7f1 View commit details
    Browse the repository at this point in the history

Commits on Jun 14, 2023

  1. Configuration menu
    Copy the full SHA
    6b15533 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    d47fa40 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    aa98ea6 View commit details
    Browse the repository at this point in the history

Commits on Jun 17, 2023

  1. Configuration menu
    Copy the full SHA
    86d53a3 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    5d46ce1 View commit details
    Browse the repository at this point in the history

Commits on Jun 20, 2023

  1. Remove flaky webgl_animation_skinning_performance from screenshot tests

    This new example takes a long time to load and can have a low framerate so the
    exact state of the animations when the screenshot is taken will be slightly
    different each time. This causes the screenshot test to be flaky.
    cstegel committed Jun 20, 2023
    Configuration menu
    Copy the full SHA
    f4cfced View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    aa34feb View commit details
    Browse the repository at this point in the history

Commits on Jul 26, 2023

  1. Configuration menu
    Copy the full SHA
    7721604 View commit details
    Browse the repository at this point in the history
  2. Restore old visuals after useLegacyLights default changed in dev

    See:
    
    1. mrdoob#26392
    2. mrdoob#26267
    
    The default behavior of light intensities was changed. Some examples had the
    light intensity values changed from 1 to 3 to preserve the old behavior.
    
    This is necessary after merging dev into this branch.
    
    The screenshots still visually look the same but I am committing them as well
    in case there were any subtle changes due to this change in lighting behavior.
    cstegel committed Jul 26, 2023
    Configuration menu
    Copy the full SHA
    4ff7701 View commit details
    Browse the repository at this point in the history
  3. Update svg_sandbox screenshot because e2e test is flaky

    I did not change any behavior related to this screenshot test.
    
    One E2E test run said it differed in 2.5% of pixels but the other
    test runs were fine. I'm updating the screenshot to hopefully
    make this test less flaky.
    cstegel committed Jul 26, 2023
    Configuration menu
    Copy the full SHA
    50a6da9 View commit details
    Browse the repository at this point in the history

Commits on Aug 7, 2023

  1. Configuration menu
    Copy the full SHA
    d66eab4 View commit details
    Browse the repository at this point in the history

Commits on Sep 11, 2023

  1. Configuration menu
    Copy the full SHA
    b5e206e View commit details
    Browse the repository at this point in the history

Commits on Oct 17, 2023

  1. Configuration menu
    Copy the full SHA
    be0b73d View commit details
    Browse the repository at this point in the history

Commits on Nov 9, 2023

  1. Configuration menu
    Copy the full SHA
    fb8a84e View commit details
    Browse the repository at this point in the history

Commits on Jan 22, 2024

  1. Configuration menu
    Copy the full SHA
    8ca2e48 View commit details
    Browse the repository at this point in the history

Commits on Apr 3, 2024

  1. Configuration menu
    Copy the full SHA
    d3aa1b4 View commit details
    Browse the repository at this point in the history
  2. Convert spaces to tabs

    I accidentally added spaces instead of tabs during a previous merge commit that required manual resolving.
    cstegel authored Apr 3, 2024
    Configuration menu
    Copy the full SHA
    607c88a View commit details
    Browse the repository at this point in the history