Motion retargetting result looks quite off

[JinwooMcLee]

Hi, first of all, thank you for sharing great project.

I'm utilizing Mixamo mocap data, to inject punch motion to my character.

This is the frame where punch hits, so skeleton's right hand is extended most, even beyond its left hand on sagittal plane.

But the joint position at the specific frame (obtained from AnimatedDrawingRig class) is somewhat off.

both shoulder (red and orange) length should be shrinked, left arm (green) should be shrinked, and right arm (yellow) should be more extended. I'm wondering how I can deal with this problem.

Edit
I guess it's due to the skeleton always projected to its sagittal plane, I'm wondering if we can set the projection plane, such as XZ plane.

Result reproduction details

• I downloaded cross punch motion from Mixamo.

• Then I imported downloaded fbx mocap file, and rotate the skeleton to do punches toward +X axis

• I exported that data to bvh format

• I edit bvh, retarget cfg, and motion cfg as follows.
  yamls_and_bvh.zip

• I generate retargeted motions.

[hjessmith]

Hi @JinwooMcLee

Thanks for the detailed bug report.

There's a view-dependent element to the pose retargeting, so it's possible that the mismatch you're seeing is due to the particular position of the camera when you took the picture of the BVH skeleton punching.

Even with the mismatch that the view-dependent retargeting could introduce, I agree with you that the poses seem pretty different. I'm investigating now to figure out why.

Edit
Just saw your edit comment. If you're sure that's the case, I won't investigate. However, you can modify the projection plane by setting the 'method' attribute of 'bvh_projection_bodypart_groups' in the retarget config. See here for more details. You can choose between 'sagittal' and 'frontal', but you cannot specify arbitrary planes yet. If you're interested in adding that functionality, I can tell you where in code to do so, and you can submit a PR if you'd like.

[JinwooMcLee]

Hi @hjessmith , I appreciate your comprehensive response.

Despite adjusting the camera's position and forward vector, the issue persists. Neither frontal plane projection for the upper body nor sagittal projection resolved the problem, as each frame displayed consistent shoulder length without any variation.

Anyways, I am keen to incorporate the XZ plane projection functionality. Could you please guide me on the next steps?

Additionally, I have three questions.

Nose becomes neck: In drawn_humanoid_pose_estimator, keypoint 0 is nose. But in here, it is annotated as neck. I am curious about the reasoning behind this particular design choice.

Motion Retargeting: How does Animated Drawings accomplish motion retargeting, particularly the transfer of motion between different body proportions? I presume this method is responsible, and this line calculates the scale factor between the motion skeleton and the character skeleton. However, the intuition behind it is unclear to me. For motion retargeting, do we sum up all the character limb lengths and all bvh skeleton lengths to compute proportions? How are discrepancies in limb proportions, such as unusually long arms or short legs, accounted for?

Projection Plane: What is the purpose of the projection plane? Could it be to incorporate a multi-perspective effect, similar to Picasso's style, as mentioned in the paper? Does setting the camera position and forward vector not achieve the same effect as establishing the projection plane? If both the camera and projection plane are set, how do they interact?
I'm wondering if there's way to get gif result without utilizing projection planes for limbs, like just watching through camera.

Thank you for your time and help.

Best regards

[hjessmith]

Nose -> Neck: Nose is the name of that particular keypoint in the coco human pose keypoint format. But we're using it as a point in the head so we can move the head around a little bit. So, in my mind, it's more akin to a neck joint than a nose joint once we're using it specifically for animation.

Motion Retargeting: Read 3.4 in the paper. It's a forward kinematics, orientation-matching method for posing the figures. The method and line you've linked are specifically for calculating how to scale the root offset of the character (i.e. when the character is walking, how far should I move the entire character left/right?). That's the only way in which differences with actor/character limb lengths are explicitly dealt with.

Projection Plane: Read 3.4 and see Figure 8.c. Projection planes are defined relative to the motion source skeleton and are used to calculate how to project the motion source's 3D limb orientations into a 2D orientation we can apply to the character. This is totally distinct from the camera position and forward vector, which are used to control the rendering camera within the scene.

If you want to change the supported projection planes, that would occur within the retargeter. Here is where the two projection planes are defined for use by the retargeter. I would do a search the code to see where these values are being used and then plan your edits accordingly. Note that they are defined relative to the motion source skeleton, where the skeleton is always assumed to be facing along the +X axis, with +Y axis corresponding to up and +Z it towards the skeleton's right side.

[JinwooMcLee]

Hi @hjessmith, I hope this message finds you well. First and foremost, I would like to express my sincere gratitude for your detailed explanation and assistance. It has been incredibly helpful.

After carefully revisiting the paper with your explanation in mind, I found that many concepts that were previously unclear to me are now much more understandable. However, I am currently facing a challenge that I would greatly appreciate your insights on.

I am working with Mixamo mocap data rendered in Blender, as shown in the image below:

I also rendered the same mocap data using Animated Drawings with sagittal projection for all limbs, as depicted in the following image:

Upon reviewing the retargeted motion, it appears that something may not be quite right. Based on my understanding, I suspect that the issue may be related to the intentional design choice of not introducing foreshortening in Animated Drawings. As a result, the Blender rendering displays foreshortening for the shoulders and other limbs, while Animated Drawings does not. Please forgive me if my assessment is incorrect.

I am eager to find a solution to this challenge and was wondering if there might be a way to introduce foreshortening or any other approach that could help address this issue.

Thank you once again for your invaluable support. I eagerly await your guidance.

[hjessmith]

You're assessment is correct. May I ask why you are trying to do this? What you're asking for is possible, but it will take a bit of time on your part to implement, as I specifically designed the system to avoid foreshortening.

I want to make sure the AD code is the best option for your use case before I try to describe how to make such a code change.

[JinwooMcLee]

Hi @hjessmith,
Thank you for your patience, as this will be a comprehensive response.

As you know, animating 2D humanoid characters is indeed a challenging task, as it usually requires drawing each frame or rigging the character and moving the rig manually.

However, AD significantly improved the user experience in animating 2D characters. AD offers motion templates, allowing the user to simply provide the character and select the desired motion.

Behind the scenes, AD does the hard works. It retargets 3D motion to the 2D character and deforms the mesh accordingly. The inclusion of a default pose detector also enhances the user experience.

Therefore, my primary needs are motion retargeting and proper 2D mesh deformation.

Last but not least, by achieving well-deformed mesh, I aim to explore whether it can be further improved with the power of diffusion models. I'm interested in applying the output from AD to all 2D humanoid characters, including, of course, children's drawings.

[hjessmith]

Thanks for your patience. I have good news and bad news. The bad news is that aI've thought about it and I can't think of a simple, straightforward way for you to modify the AD code to introduce foreshortening, which you've indicated you need. It is certainly possible but not simple.

The good news is that you can reuse many parts of AD to do what you want. The arap.py file can be used as a standalone python script, and that is what deforms the mesh based upon 'pin' positions. And the pose estimation models can be run independently of the retageting/rendering code. You'll just need to figure out how to project the motion capture joint positions onto a 2D plane in the way way that you want, then use the position of those joints on the 2D plane as 'pins' to drive the ARAP solve.

Best of luck and, if you'd like, please feel free to share any visuals create with the code.

[JinwooMcLee]

@hjessmith, Thank you sincerely for your detailed advice; it has brought much-needed clarity. I'm excited to delve into your suggestions and will certainly implement them.

To enhance my knowledge further, could you recommend any books or scholarly articles that offer a comprehensive insight into motion retargeting, especially focusing on conversions from 3D to 2D, or 2D to 2D? I realize this might be a bit off the topic, but any information on this would be extremely beneficial to my progression. I've recently read "Retargeting Motion to New Characters" by Gleicher for your reference.

I'm currently exploring computer graphics, and despite finding abundant resources, I'm encountering difficulties in locating in-depth references specifically on motion retargeting. Any suggestions you could provide would be highly appreciated.

Additionally, I'm curious whether a solid understanding of both motion retargeting and computer graphics would fulfill my requirements.

Thank you once again for your kindness and support.

Oh, and I got this character (Not mine, from this game) to walk with the power of animated drawings and other stuffs. It's far from perfect, but I guess it's a start 😊

[hjessmith]

Hello,

Motion retargeting is a fairly complex task, and I would be hesitant to suggest papers that deal with it specifically, since they will assume a lot of prior knowledge in graphics and animation.

If you are new to graphics more generally, I would recommend Fundamentals of Computer Graphics. The first few chapters about linear algebra and transforms are all useful background information: https://www.amazon.com/Fundamentals-Computer-Graphics-Steve-Marschner/dp/0367505037/ref=asc_df_0367505037/?tag=hyprod-20&linkCode=df0&hvadid=560352773205&hvpos=&hvnetw=g&hvrand=4804398781104349615&hvpone=&hvptwo=&hvqmt=&hvdev=c&hvdvcmdl=&hvlocint=&hvlocphy=9032046&hvtargid=pla-1188459326544&psc=1

Next I would recommend Parent's Animation book: https://www.amazon.com/Computer-Animation-Algorithms-Rick-Parent/dp/0124158420
It gives good information about how motion is represented in skeletal animation systems.

With both of these books read, you should have more than enough knowledge to perform 3D -> 2D retargeting.