Load morph target buffers from GLTF assets #3722

james7132 · 2022-01-19T08:57:32Z

Objective

GLTF supports storing morph target data in meshes. bevy should be able to load these buffers properly.

Solution

Add a Vec of MorphTargets to store the per-target displacements.
Extend bevy_gltf to load the associated displacements into these buffers.
Encode all morph targets as a 2D texture to support sampling more than 8 morph targets at once.

Followup

A PR for rendering the displacements should be created. This will involve the following:

Bind the texture and sampler.
Add a shader def for specializing mesh shaders for morph target support.
Include a uniform that includes the weights and indexes of the highest weighted morph targets along with a length of how many active morph targets there are.

Open Questions

What's a reasonable balance between performance and artistic flexibility? Ideally up to 256 active morph targets should be supported for maximum flexibility, but that would require 2KB of weights/indexes in every mesh uniform.
This implementation is limited to the max dimension of a texture divided by 3.Can this be avoided by using 2D texture arrays?
Will this break any future efforts to do mesh batching?
Morph target displacements are typically very small, and almost all 0. Can we save RAM and VRAM without significant loss of quality by using TextureFormat::Rgba16Float?

We already know the output ahead of time.

kodra-dev · 2023-02-17T11:50:57Z

How will morph targets and skeleton animation work together? If I'm not mistaken, usually skeleton animation is applied after the vertex positions are moved by morph targets. How are we going to achieve this wit Bevy?

james7132 · 2023-02-17T14:54:48Z

They're composable. We'll likely need to apply morph target displacements before skinning. If we move to compute shader based skinning, morph target animation can likely just be another compute shader step in the pipeline that deforms the mesh.

nicopap · 2023-03-07T12:07:08Z

For keyword searches: this feature is also known as blend shapes (Unity, Maya) or Shape Keys (blender)

nicopap · 2023-03-17T10:17:06Z

I've adopted this feature. Making progress on the morph-target branch of my bevy fork.

The idea is to use a 3D (array) texture, where each pixel is a single component of morph-target animated attribute.

james7132 · 2023-03-22T15:30:13Z

Closing in favor of #8158.

@superdump

Objective --------- - Add morph targets to `bevy_pbr` (closes bevyengine#5756) & load them from glTF - Supersedes bevyengine#3722 - Fixes bevyengine#6814 [Morph targets][1] (also known as shape interpolation, shape keys, or blend shapes) allow animating individual vertices with fine grained controls. This is typically used for facial expressions. By specifying multiple poses as vertex offset, and providing a set of weight of each pose, it is possible to define surprisingly realistic transitions between poses. Blending between multiple poses also allow composition. Morph targets are part of the [gltf standard][2] and are a feature of Unity and Unreal, and babylone.js, it is only natural to implement them in bevy. Solution -------- This implementation of morph targets uses a 3d storage texture where each pixel is a component of an animated attribute. Each layer is a different target. We use a 2d texture for each target, because the number of attribute×components×animated vertices is expected to always exceed the maximum pixel row size limit of webGL2. It copies fairly closely the way skinning is implemented on the CPU side, while on the GPU side, the shader morph target implementation is a relatively trivial detail. We add an optional `morph_texture` to the `Mesh` struct. The `morph_texture` is built through a method that accepts an iterator over attribute buffers. The `MorphWeights` component, user-accessible, controls the blend of poses used by mesh instances (so that multiple copy of the same mesh may have different weights), all the weights are uploaded to a uniform buffer of 256 `f32`. We limit to 16 poses per mesh, and a total of 256 poses. More literature: * Old babylone.js implementation (vertex attribute-based): https://www.eternalcoding.com/dev-log-1-morph-targets/ * Babylone.js implementation (similar to ours): https://www.youtube.com/watch?v=LBPRmGgU0PE * GPU gems 3: https://developer.nvidia.com/gpugems/gpugems3/part-i-geometry/chapter-3-directx-10-blend-shapes-breaking-limits * Development discord thread https://discord.com/channels/691052431525675048/1083325980615114772 https://user-images.githubusercontent.com/26321040/231181046-3bca2ab2-d4d9-472e-8098-639f1871ce2e.mp4 Acknowledgements --------------- * Thanks to @storytold for sponsoring the feature * Thanks to @superdump and @james7132 for guidance and help figuring out stuff Future work ----------- - Handling of less and more attributes (eg: animated uv, animated arbitrary attributes) - Dynamic pose allocation (so that zero-weighted poses aren't uploaded to GPU for example, enables much more total poses) - Better animation API, see bevyengine#8357 ---- Changelog --------- - Add morph targets to bevy meshes - Support up to 64 poses per mesh of individually up to 116508 vertices, animation currently strictly limited to the position, normal and tangent attributes. - Load a morph target using `Mesh::set_morph_targets` - Add `VisitMorphTargets` and `VisitMorphAttributes` traits to `bevy_render`, this allows defining morph targets (a fairly complex and nested data structure) through iterators (ie: single copy instead of passing around buffers), see documentation of those traits for details - Add `MorphWeights` component exported by `bevy_render` - `MorphWeights` control mesh's morph target weights, blending between various poses defined as morph targets. - `MorphWeights` are directly inherited by direct children (single level of hierarchy) of an entity. This allows controlling several mesh primitives through a unique entity _as per GLTF spec_. - Add `GltfMeshExtras` component to query gltf extras specific to meshes of a given node - This allows reading morph weight names, see the new `scene_viewer` `morph_viewer_plugin.rs` module for details. - Load morph targets weights and buffers in `bevy_gltf` - handle morph targets animations in `bevy_animation` (previously, it was a `warn!` log) - Add the `multiple_morph_target_meshes.gltf` asset for morph targets testing. Load it with the scene viewer. Migration Guide --------------- - (very specialized, unlikely to be touched by 3rd parties) - `MeshPipeline` now has a single `mesh_layouts` field rather than separate `mesh_layout` and `skinned_mesh_layout` fields. You should handle all possible mesh bind group layouts in your implementation. - You should also handle properly the new `MORPH_TARGETS` shader def and mesh pipeline key. A new function is exposed to make this easier: `set_mesh_binding_defs` . - The `MeshBindGroup` resource doesn't exist anymore. Mesh bind groups are computed in `queue_mesh_bind_group` system and are available as the field `GpuMesh::bind_group` . [1]: https://en.wikipedia.org/wiki/Morph_target_animation [2]: https://registry.khronos.org/glTF/specs/2.0/glTF-2.0.html#morph-targets

@superdump

Objective --------- - Add morph targets to `bevy_pbr` (closes bevyengine#5756) & load them from glTF - Supersedes bevyengine#3722 - Fixes bevyengine#6814 [Morph targets][1] (also known as shape interpolation, shape keys, or blend shapes) allow animating individual vertices with fine grained controls. This is typically used for facial expressions. By specifying multiple poses as vertex offset, and providing a set of weight of each pose, it is possible to define surprisingly realistic transitions between poses. Blending between multiple poses also allow composition. Morph targets are part of the [gltf standard][2] and are a feature of Unity and Unreal, and babylone.js, it is only natural to implement them in bevy. Solution -------- This implementation of morph targets uses a 3d storage texture where each pixel is a component of an animated attribute. Each layer is a different target. We use a 2d texture for each target, because the number of attribute×components×animated vertices is expected to always exceed the maximum pixel row size limit of webGL2. It copies fairly closely the way skinning is implemented on the CPU side, while on the GPU side, the shader morph target implementation is a relatively trivial detail. We add an optional `morph_texture` to the `Mesh` struct. The `morph_texture` is built through a method that accepts an iterator over attribute buffers. The `MorphWeights` component, user-accessible, controls the blend of poses used by mesh instances (so that multiple copy of the same mesh may have different weights), all the weights are uploaded to a uniform buffer of 256 `f32`. We limit to 16 poses per mesh, and a total of 256 poses. More literature: * Old babylone.js implementation (vertex attribute-based): https://www.eternalcoding.com/dev-log-1-morph-targets/ * Babylone.js implementation (similar to ours): https://www.youtube.com/watch?v=LBPRmGgU0PE * GPU gems 3: https://developer.nvidia.com/gpugems/gpugems3/part-i-geometry/chapter-3-directx-10-blend-shapes-breaking-limits * Development discord thread https://discord.com/channels/691052431525675048/1083325980615114772 https://user-images.githubusercontent.com/26321040/231181046-3bca2ab2-d4d9-472e-8098-639f1871ce2e.mp4 Acknowledgements --------------- * Thanks to @storytold for sponsoring the feature * Thanks to @superdump and @james7132 for guidance and help figuring out stuff Future work ----------- - Handling of less and more attributes (eg: animated uv, animated arbitrary attributes) - Dynamic pose allocation (so that zero-weighted poses aren't uploaded to GPU for example, enables much more total poses) - Better animation API, see bevyengine#8357 ---- Changelog --------- - Add morph targets to bevy meshes - Support up to 64 poses per mesh of individually up to 116508 vertices, animation currently strictly limited to the position, normal and tangent attributes. - Load a morph target using `Mesh::set_morph_targets` - Add `VisitMorphTargets` and `VisitMorphAttributes` traits to `bevy_render`, this allows defining morph targets (a fairly complex and nested data structure) through iterators (ie: single copy instead of passing around buffers), see documentation of those traits for details - Add `MorphWeights` component exported by `bevy_render` - `MorphWeights` control mesh's morph target weights, blending between various poses defined as morph targets. - `MorphWeights` are directly inherited by direct children (single level of hierarchy) of an entity. This allows controlling several mesh primitives through a unique entity _as per GLTF spec_. - Add `GltfMeshExtras` component to query gltf extras specific to meshes of a given node - This allows reading morph weight names, see the new `scene_viewer` `morph_viewer_plugin.rs` module for details. - Load morph targets weights and buffers in `bevy_gltf` - handle morph targets animations in `bevy_animation` (previously, it was a `warn!` log) - Add the `multiple_morph_target_meshes.gltf` asset for morph targets testing. Load it with the scene viewer. Migration Guide --------------- - (very specialized, unlikely to be touched by 3rd parties) - `MeshPipeline` now has a single `mesh_layouts` field rather than separate `mesh_layout` and `skinned_mesh_layout` fields. You should handle all possible mesh bind group layouts in your implementation. - You should also handle properly the new `MORPH_TARGETS` shader def and mesh pipeline key. A new function is exposed to make this easier: `set_mesh_binding_defs` . - The `MeshBindGroup` resource doesn't exist anymore. Mesh bind groups are computed in `queue_mesh_bind_group` system and are available as the field `GpuMesh::bind_group` . [1]: https://en.wikipedia.org/wiki/Morph_target_animation [2]: https://registry.khronos.org/glTF/specs/2.0/glTF-2.0.html#morph-targets

@superdump

Objective --------- - Add morph targets to `bevy_pbr` (closes bevyengine#5756) & load them from glTF - Supersedes bevyengine#3722 - Fixes bevyengine#6814 [Morph targets][1] (also known as shape interpolation, shape keys, or blend shapes) allow animating individual vertices with fine grained controls. This is typically used for facial expressions. By specifying multiple poses as vertex offset, and providing a set of weight of each pose, it is possible to define surprisingly realistic transitions between poses. Blending between multiple poses also allow composition. Morph targets are part of the [gltf standard][2] and are a feature of Unity and Unreal, and babylone.js, it is only natural to implement them in bevy. Solution -------- This implementation of morph targets uses a 3d storage texture where each pixel is a component of an animated attribute. Each layer is a different target. We use a 2d texture for each target, because the number of attribute×components×animated vertices is expected to always exceed the maximum pixel row size limit of webGL2. It copies fairly closely the way skinning is implemented on the CPU side, while on the GPU side, the shader morph target implementation is a relatively trivial detail. We add an optional `morph_texture` to the `Mesh` struct. The `morph_texture` is built through a method that accepts an iterator over attribute buffers. The `MorphWeights` component, user-accessible, controls the blend of poses used by mesh instances (so that multiple copy of the same mesh may have different weights), all the weights are uploaded to a uniform buffer of 256 `f32`. We limit to 16 poses per mesh, and a total of 256 poses. More literature: * Old babylone.js implementation (vertex attribute-based): https://www.eternalcoding.com/dev-log-1-morph-targets/ * Babylone.js implementation (similar to ours): https://www.youtube.com/watch?v=LBPRmGgU0PE * GPU gems 3: https://developer.nvidia.com/gpugems/gpugems3/part-i-geometry/chapter-3-directx-10-blend-shapes-breaking-limits * Development discord thread https://discord.com/channels/691052431525675048/1083325980615114772 https://user-images.githubusercontent.com/26321040/231181046-3bca2ab2-d4d9-472e-8098-639f1871ce2e.mp4 Acknowledgements --------------- * Thanks to @storytold for sponsoring the feature * Thanks to @superdump and @james7132 for guidance and help figuring out stuff Future work ----------- - Handling of less and more attributes (eg: animated uv, animated arbitrary attributes) - Dynamic pose allocation (so that zero-weighted poses aren't uploaded to GPU for example, enables much more total poses) - Better animation API, see bevyengine#8357 ---- Changelog --------- - Add morph targets to bevy meshes - Support up to 64 poses per mesh of individually up to 116508 vertices, animation currently strictly limited to the position, normal and tangent attributes. - Load a morph target using `Mesh::set_morph_targets` - Add `VisitMorphTargets` and `VisitMorphAttributes` traits to `bevy_render`, this allows defining morph targets (a fairly complex and nested data structure) through iterators (ie: single copy instead of passing around buffers), see documentation of those traits for details - Add `MorphWeights` component exported by `bevy_render` - `MorphWeights` control mesh's morph target weights, blending between various poses defined as morph targets. - `MorphWeights` are directly inherited by direct children (single level of hierarchy) of an entity. This allows controlling several mesh primitives through a unique entity _as per GLTF spec_. - Add `GltfMeshExtras` component to query gltf extras specific to meshes of a given node - This allows reading morph weight names, see the new `scene_viewer` `morph_viewer_plugin.rs` module for details. - Load morph targets weights and buffers in `bevy_gltf` - handle morph targets animations in `bevy_animation` (previously, it was a `warn!` log) - Add the `multiple_morph_target_meshes.gltf` asset for morph targets testing. Load it with the scene viewer. Migration Guide --------------- - (very specialized, unlikely to be touched by 3rd parties) - `MeshPipeline` now has a single `mesh_layouts` field rather than separate `mesh_layout` and `skinned_mesh_layout` fields. You should handle all possible mesh bind group layouts in your implementation. - You should also handle properly the new `MORPH_TARGETS` shader def and mesh pipeline key. A new function is exposed to make this easier: `set_mesh_binding_defs` . - The `MeshBindGroup` resource doesn't exist anymore. Mesh bind groups are computed in `queue_mesh_bind_group` system and are available as the field `GpuMesh::bind_group` . [1]: https://en.wikipedia.org/wiki/Morph_target_animation [2]: https://registry.khronos.org/glTF/specs/2.0/glTF-2.0.html#morph-targets

@superdump

Objective --------- - Add morph targets to `bevy_pbr` (closes bevyengine#5756) & load them from glTF - Supersedes bevyengine#3722 - Fixes bevyengine#6814 [Morph targets][1] (also known as shape interpolation, shape keys, or blend shapes) allow animating individual vertices with fine grained controls. This is typically used for facial expressions. By specifying multiple poses as vertex offset, and providing a set of weight of each pose, it is possible to define surprisingly realistic transitions between poses. Blending between multiple poses also allow composition. Morph targets are part of the [gltf standard][2] and are a feature of Unity and Unreal, and babylone.js, it is only natural to implement them in bevy. Solution -------- This implementation of morph targets uses a 3d storage texture where each pixel is a component of an animated attribute. Each layer is a different target. We use a 2d texture for each target, because the number of attribute×components×animated vertices is expected to always exceed the maximum pixel row size limit of webGL2. It copies fairly closely the way skinning is implemented on the CPU side, while on the GPU side, the shader morph target implementation is a relatively trivial detail. We add an optional `morph_texture` to the `Mesh` struct. The `morph_texture` is built through a method that accepts an iterator over attribute buffers. The `MorphWeights` component, user-accessible, controls the blend of poses used by mesh instances (so that multiple copy of the same mesh may have different weights), all the weights are uploaded to a uniform buffer of 256 `f32`. We limit to 16 poses per mesh, and a total of 256 poses. More literature: * Old babylone.js implementation (vertex attribute-based): https://www.eternalcoding.com/dev-log-1-morph-targets/ * Babylone.js implementation (similar to ours): https://www.youtube.com/watch?v=LBPRmGgU0PE * GPU gems 3: https://developer.nvidia.com/gpugems/gpugems3/part-i-geometry/chapter-3-directx-10-blend-shapes-breaking-limits * Development discord thread https://discord.com/channels/691052431525675048/1083325980615114772 https://user-images.githubusercontent.com/26321040/231181046-3bca2ab2-d4d9-472e-8098-639f1871ce2e.mp4 Acknowledgements --------------- * Thanks to @storytold for sponsoring the feature * Thanks to @superdump and @james7132 for guidance and help figuring out stuff Future work ----------- - Handling of less and more attributes (eg: animated uv, animated arbitrary attributes) - Dynamic pose allocation (so that zero-weighted poses aren't uploaded to GPU for example, enables much more total poses) - Better animation API, see bevyengine#8357 ---- Changelog --------- - Add morph targets to bevy meshes - Support up to 64 poses per mesh of individually up to 116508 vertices, animation currently strictly limited to the position, normal and tangent attributes. - Load a morph target using `Mesh::set_morph_targets` - Add `VisitMorphTargets` and `VisitMorphAttributes` traits to `bevy_render`, this allows defining morph targets (a fairly complex and nested data structure) through iterators (ie: single copy instead of passing around buffers), see documentation of those traits for details - Add `MorphWeights` component exported by `bevy_render` - `MorphWeights` control mesh's morph target weights, blending between various poses defined as morph targets. - `MorphWeights` are directly inherited by direct children (single level of hierarchy) of an entity. This allows controlling several mesh primitives through a unique entity _as per GLTF spec_. - Add `GltfMeshExtras` component to query gltf extras specific to meshes of a given node - This allows reading morph weight names, see the new `scene_viewer` `morph_viewer_plugin.rs` module for details. - Load morph targets weights and buffers in `bevy_gltf` - handle morph targets animations in `bevy_animation` (previously, it was a `warn!` log) - Add the `multiple_morph_target_meshes.gltf` asset for morph targets testing. Load it with the scene viewer. Migration Guide --------------- - (very specialized, unlikely to be touched by 3rd parties) - `MeshPipeline` now has a single `mesh_layouts` field rather than separate `mesh_layout` and `skinned_mesh_layout` fields. You should handle all possible mesh bind group layouts in your implementation. - You should also handle properly the new `MORPH_TARGETS` shader def and mesh pipeline key. A new function is exposed to make this easier: `set_mesh_binding_defs` . - The `MeshBindGroup` resource doesn't exist anymore. Mesh bind groups are computed in `queue_mesh_bind_group` system and are available as the field `GpuMesh::bind_group` . [1]: https://en.wikipedia.org/wiki/Morph_target_animation [2]: https://registry.khronos.org/glTF/specs/2.0/glTF-2.0.html#morph-targets

@superdump

Objective --------- - Add morph targets to `bevy_pbr` (closes bevyengine#5756) & load them from glTF - Supersedes bevyengine#3722 - Fixes bevyengine#6814 [Morph targets][1] (also known as shape interpolation, shape keys, or blend shapes) allow animating individual vertices with fine grained controls. This is typically used for facial expressions. By specifying multiple poses as vertex offset, and providing a set of weight of each pose, it is possible to define surprisingly realistic transitions between poses. Blending between multiple poses also allow composition. Morph targets are part of the [gltf standard][2] and are a feature of Unity and Unreal, and babylone.js, it is only natural to implement them in bevy. Solution -------- This implementation of morph targets uses a 3d storage texture where each pixel is a component of an animated attribute. Each layer is a different target. We use a 2d texture for each target, because the number of attribute×components×animated vertices is expected to always exceed the maximum pixel row size limit of webGL2. It copies fairly closely the way skinning is implemented on the CPU side, while on the GPU side, the shader morph target implementation is a relatively trivial detail. We add an optional `morph_texture` to the `Mesh` struct. The `morph_texture` is built through a method that accepts an iterator over attribute buffers. The `MorphWeights` component, user-accessible, controls the blend of poses used by mesh instances (so that multiple copy of the same mesh may have different weights), all the weights are uploaded to a uniform buffer of 256 `f32`. We limit to 16 poses per mesh, and a total of 256 poses. More literature: * Old babylone.js implementation (vertex attribute-based): https://www.eternalcoding.com/dev-log-1-morph-targets/ * Babylone.js implementation (similar to ours): https://www.youtube.com/watch?v=LBPRmGgU0PE * GPU gems 3: https://developer.nvidia.com/gpugems/gpugems3/part-i-geometry/chapter-3-directx-10-blend-shapes-breaking-limits * Development discord thread https://discord.com/channels/691052431525675048/1083325980615114772 https://user-images.githubusercontent.com/26321040/231181046-3bca2ab2-d4d9-472e-8098-639f1871ce2e.mp4 Acknowledgements --------------- * Thanks to @storytold for sponsoring the feature * Thanks to @superdump and @james7132 for guidance and help figuring out stuff Future work ----------- - Handling of less and more attributes (eg: animated uv, animated arbitrary attributes) - Dynamic pose allocation (so that zero-weighted poses aren't uploaded to GPU for example, enables much more total poses) - Better animation API, see bevyengine#8357 ---- Changelog --------- - Add morph targets to bevy meshes - Support up to 64 poses per mesh of individually up to 116508 vertices, animation currently strictly limited to the position, normal and tangent attributes. - Load a morph target using `Mesh::set_morph_targets` - Add `VisitMorphTargets` and `VisitMorphAttributes` traits to `bevy_render`, this allows defining morph targets (a fairly complex and nested data structure) through iterators (ie: single copy instead of passing around buffers), see documentation of those traits for details - Add `MorphWeights` component exported by `bevy_render` - `MorphWeights` control mesh's morph target weights, blending between various poses defined as morph targets. - `MorphWeights` are directly inherited by direct children (single level of hierarchy) of an entity. This allows controlling several mesh primitives through a unique entity _as per GLTF spec_. - Add `GltfMeshExtras` component to query gltf extras specific to meshes of a given node - This allows reading morph weight names, see the new `scene_viewer` `morph_viewer_plugin.rs` module for details. - Load morph targets weights and buffers in `bevy_gltf` - handle morph targets animations in `bevy_animation` (previously, it was a `warn!` log) - Add the `multiple_morph_target_meshes.gltf` asset for morph targets testing. Load it with the scene viewer. Migration Guide --------------- - (very specialized, unlikely to be touched by 3rd parties) - `MeshPipeline` now has a single `mesh_layouts` field rather than separate `mesh_layout` and `skinned_mesh_layout` fields. You should handle all possible mesh bind group layouts in your implementation. - You should also handle properly the new `MORPH_TARGETS` shader def and mesh pipeline key. A new function is exposed to make this easier: `set_mesh_binding_defs` . - The `MeshBindGroup` resource doesn't exist anymore. Mesh bind groups are computed in `queue_mesh_bind_group` system and are available as the field `GpuMesh::bind_group` . [1]: https://en.wikipedia.org/wiki/Morph_target_animation [2]: https://registry.khronos.org/glTF/specs/2.0/glTF-2.0.html#morph-targets

@superdump

Objective --------- - Add morph targets to `bevy_pbr` (closes bevyengine#5756) & load them from glTF - Supersedes bevyengine#3722 - Fixes bevyengine#6814 [Morph targets][1] (also known as shape interpolation, shape keys, or blend shapes) allow animating individual vertices with fine grained controls. This is typically used for facial expressions. By specifying multiple poses as vertex offset, and providing a set of weight of each pose, it is possible to define surprisingly realistic transitions between poses. Blending between multiple poses also allow composition. Morph targets are part of the [gltf standard][2] and are a feature of Unity and Unreal, and babylone.js, it is only natural to implement them in bevy. Solution -------- This implementation of morph targets uses a 3d storage texture where each pixel is a component of an animated attribute. Each layer is a different target. We use a 2d texture for each target, because the number of attribute×components×animated vertices is expected to always exceed the maximum pixel row size limit of webGL2. It copies fairly closely the way skinning is implemented on the CPU side, while on the GPU side, the shader morph target implementation is a relatively trivial detail. We add an optional `morph_texture` to the `Mesh` struct. The `morph_texture` is built through a method that accepts an iterator over attribute buffers. The `MorphWeights` component, user-accessible, controls the blend of poses used by mesh instances (so that multiple copy of the same mesh may have different weights), all the weights are uploaded to a uniform buffer of 256 `f32`. We limit to 16 poses per mesh, and a total of 256 poses. More literature: * Old babylone.js implementation (vertex attribute-based): https://www.eternalcoding.com/dev-log-1-morph-targets/ * Babylone.js implementation (similar to ours): https://www.youtube.com/watch?v=LBPRmGgU0PE * GPU gems 3: https://developer.nvidia.com/gpugems/gpugems3/part-i-geometry/chapter-3-directx-10-blend-shapes-breaking-limits * Development discord thread https://discord.com/channels/691052431525675048/1083325980615114772 https://user-images.githubusercontent.com/26321040/231181046-3bca2ab2-d4d9-472e-8098-639f1871ce2e.mp4 Acknowledgements --------------- * Thanks to @storytold for sponsoring the feature * Thanks to @superdump and @james7132 for guidance and help figuring out stuff Future work ----------- - Handling of less and more attributes (eg: animated uv, animated arbitrary attributes) - Dynamic pose allocation (so that zero-weighted poses aren't uploaded to GPU for example, enables much more total poses) - Better animation API, see bevyengine#8357 ---- Changelog --------- - Add morph targets to bevy meshes - Support up to 64 poses per mesh of individually up to 116508 vertices, animation currently strictly limited to the position, normal and tangent attributes. - Load a morph target using `Mesh::set_morph_targets` - Add `VisitMorphTargets` and `VisitMorphAttributes` traits to `bevy_render`, this allows defining morph targets (a fairly complex and nested data structure) through iterators (ie: single copy instead of passing around buffers), see documentation of those traits for details - Add `MorphWeights` component exported by `bevy_render` - `MorphWeights` control mesh's morph target weights, blending between various poses defined as morph targets. - `MorphWeights` are directly inherited by direct children (single level of hierarchy) of an entity. This allows controlling several mesh primitives through a unique entity _as per GLTF spec_. - Add `GltfMeshExtras` component to query gltf extras specific to meshes of a given node - This allows reading morph weight names, see the new `scene_viewer` `morph_viewer_plugin.rs` module for details. - Load morph targets weights and buffers in `bevy_gltf` - handle morph targets animations in `bevy_animation` (previously, it was a `warn!` log) - Add the `multiple_morph_target_meshes.gltf` asset for morph targets testing. Load it with the scene viewer. Migration Guide --------------- - (very specialized, unlikely to be touched by 3rd parties) - `MeshPipeline` now has a single `mesh_layouts` field rather than separate `mesh_layout` and `skinned_mesh_layout` fields. You should handle all possible mesh bind group layouts in your implementation. - You should also handle properly the new `MORPH_TARGETS` shader def and mesh pipeline key. A new function is exposed to make this easier: `set_mesh_binding_defs` . - The `MeshBindGroup` resource doesn't exist anymore. Mesh bind groups are computed in `queue_mesh_bind_group` system and are available as the field `GpuMesh::bind_group` . [1]: https://en.wikipedia.org/wiki/Morph_target_animation [2]: https://registry.khronos.org/glTF/specs/2.0/glTF-2.0.html#morph-targets

# Objective - Add morph targets to `bevy_pbr` (closes #5756) & load them from glTF - Supersedes #3722 - Fixes #6814 [Morph targets][1] (also known as shape interpolation, shape keys, or blend shapes) allow animating individual vertices with fine grained controls. This is typically used for facial expressions. By specifying multiple poses as vertex offset, and providing a set of weight of each pose, it is possible to define surprisingly realistic transitions between poses. Blending between multiple poses also allow composition. Morph targets are part of the [gltf standard][2] and are a feature of Unity and Unreal, and babylone.js, it is only natural to implement them in bevy. ## Solution This implementation of morph targets uses a 3d texture where each pixel is a component of an animated attribute. Each layer is a different target. We use a 2d texture for each target, because the number of attribute×components×animated vertices is expected to always exceed the maximum pixel row size limit of webGL2. It copies fairly closely the way skinning is implemented on the CPU side, while on the GPU side, the shader morph target implementation is a relatively trivial detail. We add an optional `morph_texture` to the `Mesh` struct. The `morph_texture` is built through a method that accepts an iterator over attribute buffers. The `MorphWeights` component, user-accessible, controls the blend of poses used by mesh instances (so that multiple copy of the same mesh may have different weights), all the weights are uploaded to a uniform buffer of 256 `f32`. We limit to 16 poses per mesh, and a total of 256 poses. More literature: * Old babylone.js implementation (vertex attribute-based): https://www.eternalcoding.com/dev-log-1-morph-targets/ * Babylone.js implementation (similar to ours): https://www.youtube.com/watch?v=LBPRmGgU0PE * GPU gems 3: https://developer.nvidia.com/gpugems/gpugems3/part-i-geometry/chapter-3-directx-10-blend-shapes-breaking-limits * Development discord thread https://discord.com/channels/691052431525675048/1083325980615114772 https://user-images.githubusercontent.com/26321040/231181046-3bca2ab2-d4d9-472e-8098-639f1871ce2e.mp4 https://github.com/bevyengine/bevy/assets/26321040/d2a0c544-0ef8-45cf-9f99-8c3792f5a258 ## Acknowledgements * Thanks to `storytold` for sponsoring the feature * Thanks to `superdump` and `james7132` for guidance and help figuring out stuff ## Future work - Handling of less and more attributes (eg: animated uv, animated arbitrary attributes) - Dynamic pose allocation (so that zero-weighted poses aren't uploaded to GPU for example, enables much more total poses) - Better animation API, see #8357 ---- ## Changelog - Add morph targets to bevy meshes - Support up to 64 poses per mesh of individually up to 116508 vertices, animation currently strictly limited to the position, normal and tangent attributes. - Load a morph target using `Mesh::set_morph_targets` - Add `VisitMorphTargets` and `VisitMorphAttributes` traits to `bevy_render`, this allows defining morph targets (a fairly complex and nested data structure) through iterators (ie: single copy instead of passing around buffers), see documentation of those traits for details - Add `MorphWeights` component exported by `bevy_render` - `MorphWeights` control mesh's morph target weights, blending between various poses defined as morph targets. - `MorphWeights` are directly inherited by direct children (single level of hierarchy) of an entity. This allows controlling several mesh primitives through a unique entity _as per GLTF spec_. - Add `MorphTargetNames` component, naming each indices of loaded morph targets. - Load morph targets weights and buffers in `bevy_gltf` - handle morph targets animations in `bevy_animation` (previously, it was a `warn!` log) - Add the `MorphStressTest.gltf` asset for morph targets testing, taken from the glTF samples repo, CC0. - Add morph target manipulation to `scene_viewer` - Separate the animation code in `scene_viewer` from the rest of the code, reducing `#[cfg(feature)]` noise - Add the `morph_targets.rs` example to show off how to manipulate morph targets, loading `MorpStressTest.gltf` ## Migration Guide - (very specialized, unlikely to be touched by 3rd parties) - `MeshPipeline` now has a single `mesh_layouts` field rather than separate `mesh_layout` and `skinned_mesh_layout` fields. You should handle all possible mesh bind group layouts in your implementation - You should also handle properly the new `MORPH_TARGETS` shader def and mesh pipeline key. A new function is exposed to make this easier: `setup_moprh_and_skinning_defs` - The `MeshBindGroup` is now `MeshBindGroups`, cached bind groups are now accessed through the `get` method. [1]: https://en.wikipedia.org/wiki/Morph_target_animation [2]: https://registry.khronos.org/glTF/specs/2.0/glTF-2.0.html#morph-targets --------- Co-authored-by: François <mockersf@gmail.com> Co-authored-by: Carter Anderson <mcanders1@gmail.com>

Load morph targets from GLTF assets

e164a10

github-actions bot added the S-Needs-Triage This issue needs to be labelled label Jan 19, 2022

james7132 added A-Assets Load files from disk to use for things like images, models, and sounds A-Rendering Drawing game state to the screen C-Enhancement A new feature and removed S-Needs-Triage This issue needs to be labelled labels Jan 19, 2022

james7132 marked this pull request as ready for review January 19, 2022 08:58

james7132 added 12 commits January 19, 2022 01:13

Fix docs

a75834c

Avoid maps to remove space and time overheads

e553f16

We already know the output ahead of time.

Fix docs

0ee7a27

Implement GPU submit for morph targets

f2f2d2a

Move to dedicated morph target module

1d311c4

Appease clippy

4fe4bea

Use a texture to store the morph target data

054cfe1

Fix formatting

53703c7

Fix docs

160aab3

Appease clippy

862057f

Fix and optimize strongest_n

de8a566

Fix format

12acdf2

mockersf mentioned this pull request Jan 24, 2022

[Merged by Bors] - Gltf animations #3751

Closed

james7132 mentioned this pull request Feb 23, 2022

Tracking Issue: MVP Animation #4026

Open

14 tasks

james7132 added 2 commits March 22, 2022 21:33

Merge branch 'main' into morph-targets

31f0e88

Prepare for a compute shader approach for morph targets

a442268

Weibye added the S-Adopt-Me The original PR author has no intent to complete this work. Pick me up! label Aug 10, 2022

willstott101 mentioned this pull request Aug 21, 2022

GLTF Morph Targets #5756

Closed

nicopap removed the S-Adopt-Me The original PR author has no intent to complete this work. Pick me up! label Mar 17, 2023

nicopap mentioned this pull request Mar 22, 2023

Add morph targets #8158

Merged

james7132 closed this Mar 22, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Load morph target buffers from GLTF assets #3722

Load morph target buffers from GLTF assets #3722

james7132 commented Jan 19, 2022 •

edited

Loading

kodra-dev commented Feb 17, 2023

james7132 commented Feb 17, 2023

nicopap commented Mar 7, 2023

nicopap commented Mar 17, 2023

james7132 commented Mar 22, 2023

Load morph target buffers from GLTF assets #3722

Load morph target buffers from GLTF assets #3722

Conversation

james7132 commented Jan 19, 2022 • edited Loading

Objective

Solution

Followup

Open Questions

kodra-dev commented Feb 17, 2023

james7132 commented Feb 17, 2023

nicopap commented Mar 7, 2023

nicopap commented Mar 17, 2023

james7132 commented Mar 22, 2023

james7132 commented Jan 19, 2022 •

edited

Loading