Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add skinning support for gfx-replay and legacy replay rendering. #2120

Merged
merged 25 commits into from
Oct 5, 2023

Conversation

0mdc
Copy link
Contributor

@0mdc 0mdc commented May 31, 2023

Motivation and Context

This changeset adds support for:

  • Skinned mesh gfx-replay support
  • Classic replay rendering of skinned meshes

Capture from the SIRo sandbox tool, which renders using the classic replay renderer:

fetch-2023-09-29_19.20.31.mp4

How it works

Overview

This changeset introduces a new structure, Rig, which contains a node for each bone of a skinned model. During asset instantiation, the Rig is associated to the model instance so that it can be posed during rendering.

Therefore, the skinned model is animated by modifying transforms of the Rig bone nodes. This is currently done in either of those ways:

  • via articulated objects: Bone nodes are parented to their matching articulated object links.
  • via replay rendering: Bone node world transforms are written from the rigUpdate.pose in gfx-replay keyframes.

Gfx-replay

Two new keyframe fields are introduced:

  • rigCreations: Registers a new rig, along with its bone names.
  • rigUpdates: Lists the new transforms for each bone (pose).

Additionally, RenderAssetCreationInfo now has a rigId field.

How Has This Been Tested

  • Unit test
  • Tested in Python viewer, Blender and SIRo Sandbox Tool

Types of changes

  • Docs change / refactoring / dependency upgrade
  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)

Checklist

  • My code follows the code style of this project.
  • My change requires a change to the documentation.
  • I have updated the documentation accordingly.
  • I have read the CONTRIBUTING document.
  • I have completed my CLA (see CONTRIBUTING)
  • I have added tests to cover my changes.
  • All new and existing tests passed.

@0mdc 0mdc requested a review from eundersander May 31, 2023 18:30
@facebook-github-bot facebook-github-bot added the CLA Signed Do not delete this pull request or issue due to inactivity. label May 31, 2023
@0mdc 0mdc requested review from jturner65 and aclegg3 June 5, 2023 16:14
Copy link
Contributor

@eundersander eundersander left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good progress! I left comments. Let me know when I should look again!

src/esp/assets/ResourceManager.cpp Outdated Show resolved Hide resolved
src/esp/assets/ResourceManager.cpp Show resolved Hide resolved
src/esp/assets/ResourceManager.cpp Outdated Show resolved Hide resolved
src/esp/gfx/replay/Player.cpp Outdated Show resolved Hide resolved
src/esp/gfx/replay/Player.cpp Outdated Show resolved Hide resolved
src/esp/gfx/replay/Recorder.cpp Outdated Show resolved Hide resolved
@@ -150,9 +158,9 @@ bool fromJsonValue(const JsonGenericValue& obj,
metadata::attributes::ObjectInstanceShaderType& x) {
std::string shaderTypeToUseString;
// read as string
bool shaderTypeSucceess = fromJsonValue(obj, shaderTypeToUseString);
bool shaderTypeSuccess = fromJsonValue(obj, shaderTypeToUseString);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for fixing all the spelling typos!

src/esp/gfx/replay/Keyframe.h Outdated Show resolved Hide resolved
@@ -945,5 +938,40 @@ void BulletPhysicsManager::removeRigidConstraint(int constraintId) {
}
}

void BulletPhysicsManager::instantiateSkinnedModel(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@aclegg3 maybe you wanna take a look at changes to BulletPhysicsManager.

src/esp/gfx/replay/Recorder.h Outdated Show resolved Hide resolved
@0mdc 0mdc marked this pull request as ready for review September 10, 2023 23:19
/**
* @brief The next available unique ID for instantiated rigs.
*/
int _nextRigInstanceID = 0;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the future we could revisit this so that it managed IDs similar to how PhysicsManager does, reusing freed IDs instead of just incrementing, on the off chance that the RigManager may persist through billions of training cycles.

Copy link
Contributor

@jturner65 jturner65 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Always nice to see some things move out of ResourceManager that can.

@0mdc
Copy link
Contributor Author

0mdc commented Oct 3, 2023

FYI - Tests were passing prior to rebasing to main due to our CI image losing support.

Job was rejected because resource class gpu.nvidia.medium, image ubuntu-2004-cuda-11.4:202110-01 is not a valid resource class

The last run was green.

Copy link
Contributor

@eundersander eundersander left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@0mdc 0mdc merged commit b6b2d83 into main Oct 5, 2023
10 checks passed
@0mdc 0mdc deleted the skinning-gfx-replay branch October 5, 2023 14:45
@0mdc 0mdc restored the skinning-gfx-replay branch November 10, 2023 17:37
@0mdc 0mdc deleted the skinning-gfx-replay branch November 10, 2023 19:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
CLA Signed Do not delete this pull request or issue due to inactivity.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants