Animate model by pose parameters

[ZWhitey]

Hi @mkocabas
I'm trying to use pose parameters to animate my model in 3D engine. I used SMPL model and apply all pose parameters to my model, but my model got some strange rotation.

These results is produced by your demo video.
Every motion looks normally, only problem is that strange rotation.
Do you have any idea how to deal with this problem?

[samuelmorais]

@ZWhitey Could you please share how did you apply the joints3D coordinates to the joints?

I used this repository that reads files with openpose body parts reference and apply it with Inverse Kinematics: https://github.com/keel-210/OpenPose-Rig

But I think you have a more direct way of applying the joint3D data to the joints.

Anyway, using OpenPose-Rig I am having the same problem you observed.

I am also looking for an answer to this issue.

[ZWhitey]

@samuelmorais I rotated every rigs directly by pose data. I didn't use joints3D coordinates.

[mkocabas]

Normally, SMPL bodies are rotated 180 degrees on the x-axis (meshes are upside down). Rotating the mesh 180 degrees should solve your problem as I did during rendering: https://github.com/mkocabas/VIBE/blob/master/lib/utils/renderer.py#L82

[ZWhitey]

Thanks for your answer, I still got one problem.
As second and third image I showed, that mesh is rotated about 90 degrees, Is there any way to solve this problem?
I tried to lock the rotation of hip on x and z axis to prevent that problem, but I'm not sure whether this is a good way to do.

[acoMCMXCVI]

Hey @ZWhitey!

@samuelmorais I rotated every rigs directly by pose data. I didn't use joints3D coordinates.

Can you please explain how you use pose data to get pose in engine? Did you use SMPL plugin or manually add rotations to every joint in engine, or something else?

Thank you

[ZWhitey]

@acoMCMXCVI I just rotate every joints in unity manually

[acoMCMXCVI]

@ZWhitey Thank you for your answer. Did you convert axis angle rotation to matrix or something like that using Rodrigues' rotation formula? I have problem how to use axis angle rotation because i need some other rotation representation. So i am not sure which conversion works.

Thank you

[pekopull]

@ZWhitey Hello, can you show the code on how to rotate every joints in unity manually? Thank you.

[pekopull]

I figured out how to rotate in unity.

I use Rodrigues to convert Axis-angle format (n_frames, 24, 3) to rotation matrix format (n_frames, 24, 3, 3) first. And then convert blender coordinate system to unity coordinate system.

              SimpleJSON.JSONNode pose = SimpleJSON.JSON.Parse (OutputJSON.text);

              Quaternion quat = Quaternion.LookRotation(
                  
                  new Vector3(-pose[index][0][2].AsFloat, pose[index][1][2].AsFloat, pose[index][2][2].AsFloat),
                  new Vector3(-pose[index][0][1].AsFloat, pose[index][1][1].AsFloat, pose[index][2][1].AsFloat)
              );
             
		/*	Quaternions */
		_bones[i].localRotation = quat;

              if (boneName == "Pelvis"){
                 _bones[i].Rotate(new Vector3(180, 0, 0), Space.World);
              }

Reference:

VIBE/lib/utils/fbx_output.py

Line 111 in 03ed50d

def process_pose(current_frame, pose, trans, pelvis_position):

https://stackoverflow.com/questions/1263072/changing-a-matrix-from-right-handed-to-left-handed-coordinate-system
https://stackoverflow.com/a/53452698/10242733

[pekopull]

Hello, you can get the Rodriguez formula here. https://github.com/mkocabas/VIBE/blob/master/lib/utils/fbx_output.py#L81 It converts Axis-angle format (n_frames, 24, 3) to rotation matrix format (n_frames, 24, 3, 3) Furthermore, if you want to use it in unity, you need to convert blender coordinate system to unity coordinate system. Like the code below: Quaternion quat = Quaternion.LookRotation( new Vector3(-pose[index][0][2].AsFloat, pose[index][1][2].AsFloat, pose[index][2][2].AsFloat), new Vector3(-pose[index][0][1].AsFloat, pose[index][1][1].AsFloat, pose[index][2][1].AsFloat) ); Some reference here: https://github.com/mkocabas/VIBE/blob/03ed50d3121c37a4abfcc25dc995f08b63fd7f1f/lib/utils/fbx_output.py#L111 https://stackoverflow.com/questions/1263072/changing-a-matrix-from-right-handed-to-left-handed-coordinate-system https://stackoverflow.com/questions/53447104/how-to-apply-transformation-using-3x3-rotation-matrix-and-a-translation-vector/53452698#53452698 Denis Braslavskiy ***@***.***> 於 2022年10月8日 週六 下午6:30寫道：

…

@pekopull <https://github.com/pekopull> Hi! Could you share the complete code, please? Especially interested in the transformation according to the Rodriguez formula, I implemented this method, but the result is clearly not what is needed — Reply to this email directly, view it on GitHub <#26 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AG5HQ6GVWOFLI3K2AIXBZKLWCFEMPANCNFSM4KS7JRYA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[BrasD99]

Dear @pekopull,

I would like to write a small thank you for the help in solving the problem. It was thanks to your comment that i managed to transfer the person pose to the SMPL model in Unity.

Now i have finally switched to PARE model, developed by @mkocabas. For your information, it already allows to output predictions for pose in the form of rotation matrix (n_frames, 24, 3, 3). This greatly simplifies the calculations on the C# side, there is no need to do calculations by Rodriguez formula (thanks to @mkocabas).

For developers who may encounter this problem, i attach a small snippet of code :)

private static readonly string[] BoneNames = new string[]
{
    "Pelvis",
    "L_Hip",
    "R_Hip",
    "Spine1",
    "L_Knee",
    "R_Knee",
    "Spine2",
    "L_Ankle",
    "R_Ankle",
    "Spine3",
    "L_Foot",
    "R_Foot",
    "Neck",
    "L_Collar",
    "R_Collar",
    "Head",
    "L_Shoulder",
    "R_Shoulder",
    "L_Elbow",
    "R_Elbow",
    "L_Wrist",
    "R_Wrist",
    "L_Hand",
    "R_Hand"
];

var player = GameObject.Find("Player");

/* Here is your logic of reading and parsing data from the dump (to playerDump variable)
...
*/

// playerDump.Body.Pose has type float[][][]
UpdateModel(player, playerDump.Body.Pose);

private static void UpdateModel(GameObject player, float[][][] pose)
{
    for (int boneIndex = 0; boneIndex < pose.Length; boneIndex++)
    {
        var boneName = $"m_avg_{BoneNames[boneIndex]}";
        var objectBone = FindChildGameObject(player, boneName);

        var quat = Quaternion.LookRotation(
            new Vector3(-pose[boneIndex][0][2], pose[boneIndex][1][2], pose[boneIndex][2][2]),
            new Vector3(-pose[boneIndex][0][1], pose[boneIndex][1][1], pose[boneIndex][2][1])
        );

        if (boneIndex == 0)
        {
            var quat_x = Quaternion.AngleAxis(-90, new Vector3(1, 0, 0));
            var quat_z = Quaternion.AngleAxis(-90, new Vector3(0, 0, 1));
            var r = Quaternion.Euler(180, 0, 0) * quat;
            objectBone.transform.localRotation = r;
        }
        else
        {
            objectBone.transform.localRotation = quat;
        }
    }
}

private static GameObject FindChildGameObject(GameObject parent, string name)
{
    var children = parent.GetComponentsInChildren<Transform>();
    return children.FirstOrDefault(u => u.gameObject.name == name)?.gameObject;
}

Hope it helps!

And i also give the results of displaying pose.

Source image	Destination model pose