MediaPipe Pose - mapping to SMPL model

[BrasD99]

Please tell me how it is possible to map the output values of the model to the SMPL model? So far, all my attempts have been unsuccessful...

[kuaashish]

HI @BrasD99,
Would you please elaborate the issue with the complete details, exact use case as per the requirement(realtime example), platform such as Python, C++, Android or other solutions available all will be useful to us to provide the possible solution in the support you looking from us. Thank you!

[BrasD99]

Hi @kuaashish,
I use MediaPipe Pose (BlazePose) in my Python project to process video frames. Then I looked at the description of the points that the model outputs, as well as the points on the SMPL model:


In the next step, I'm trying to map your points from the output of your model to SMPL, for this I have such a dictionary:
mp_to_smpl_mapper = {
32: 10, #L_Toe
31: 11, #R_Toe
28: 7, #L_Ankle
27: 8, #R_Ankle
26: 4, #L_Knee
25: 5, #R_Knee
24: 1, #L_Hip
23: 2, #R_Hip
12: 16, #L_Shoulder
11: 17, #R_Shoulder
14: 18, #L_Elbow,
13: 19, #R_Elbow
16: 20, #L_Wrist
15: 21, #R_Wrist
0: 15, #Head
}
Ultimately, my task is to display a person's pose in Unity3D, so the task boils down to giving names to each point and finding a body part by name in the SMPL model. Here is an example of data processing for Unity (these arrays were actually obtained from your model)
var bodyParts = new Dictionary<string, double[]>
{
{"m_avg_L_Toe", new double[3]{ 0.11499792, 0.95247597, 0.42436841 } },
{"m_avg_R_Toe", new double[3]{ 0.8176074, 0.96950567, 0.88876796 } },
{"m_avg_L_Ankle", new double[3]{ 0.23537651, 0.87956297, 0.97798741 } },
{"m_avg_R_Ankle", new double[3]{ 0.77714473, 0.90175259, 1.41494441 } },
{"m_avg_L_Knee", new double[3]{ 0.41185567, 0.71771574, 0.08711531 } },
{"m_avg_R_Knee", new double[3]{ 0.81979096, 0.73399705, 0.35799798 } },
{"m_avg_L_Hip", new double[3]{ 0.56988579, 0.51334482, -0.03835907 } },
{"m_avg_R_Hip", new double[3]{ 0.81553215, 0.51021743, 0.03776015 } },
{"m_avg_L_Shoulder", new double[3]{ 0.46652973, 0.20263949, -0.88932508 } },
{"m_avg_R_Shoulder", new double[3]{ 0.89907336, 0.20947775, -0.78829497 } },
{"m_avg_L_Elbow", new double[3]{ 0.42310753, 0.3573367, -0.9505372 } },
{"m_avg_R_Elbow", new double[3]{ 0.9305408, 0.37685162, -0.62403321 } },
{"m_avg_L_Wrist", new double[3]{ 0.44996896, 0.49648461, -1.49405789 } },
{"m_avg_R_Wrist", new double[3]{ 0.83942693, 0.51275945, -0.99922866 } },
{"m_avg_Head", new double[3]{ 0.67040175, 0.1112375, -1.85335028 } },
};
As a result of applying points to parts of the SMPL model, I get an unpredictable result:

I also tried to visualize the application of points to the SMPL model using the Smplx library in a python project, but the results are the same, the model assumes an incorrect pose. Most likely, I'm missing something, because when visualizing the skeleton through your code, it seems that it is correct...

[kuaashish]

@BrasD99,
we have already similar thread closed #3790 here, please go through the discussion and let us know if still you need the further help. Thank you!

[google-ml-butler]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you.

[google-ml-butler]

Closing as stale. Please reopen if you'd like to work on this further.

[google-ml-butler]

Are you satisfied with the resolution of your issue?
Yes
No

[matlab2017]

Hi @kuaashish, I use MediaPipe Pose (BlazePose) in my Python project to process video frames. Then I looked at the description of the points that the model outputs, as well as the points on the SMPL model: In the next step, I'm trying to map your points from the output of your model to SMPL, for this I have such a dictionary: mp_to_smpl_mapper = { 32: 10, #L_Toe 31: 11, #R_Toe 28: 7, #L_Ankle 27: 8, #R_Ankle 26: 4, #L_Knee 25: 5, #R_Knee 24: 1, #L_Hip 23: 2, #R_Hip 12: 16, #L_Shoulder 11: 17, #R_Shoulder 14: 18, #L_Elbow, 13: 19, #R_Elbow 16: 20, #L_Wrist 15: 21, #R_Wrist 0: 15, #Head } Ultimately, my task is to display a person's pose in Unity3D, so the task boils down to giving names to each point and finding a body part by name in the SMPL model. Here is an example of data processing for Unity (these arrays were actually obtained from your model) var bodyParts = new Dictionary<string, double[]> { {"m_avg_L_Toe", new double[3]{ 0.11499792, 0.95247597, 0.42436841 } }, {"m_avg_R_Toe", new double[3]{ 0.8176074, 0.96950567, 0.88876796 } }, {"m_avg_L_Ankle", new double[3]{ 0.23537651, 0.87956297, 0.97798741 } }, {"m_avg_R_Ankle", new double[3]{ 0.77714473, 0.90175259, 1.41494441 } }, {"m_avg_L_Knee", new double[3]{ 0.41185567, 0.71771574, 0.08711531 } }, {"m_avg_R_Knee", new double[3]{ 0.81979096, 0.73399705, 0.35799798 } }, {"m_avg_L_Hip", new double[3]{ 0.56988579, 0.51334482, -0.03835907 } }, {"m_avg_R_Hip", new double[3]{ 0.81553215, 0.51021743, 0.03776015 } }, {"m_avg_L_Shoulder", new double[3]{ 0.46652973, 0.20263949, -0.88932508 } }, {"m_avg_R_Shoulder", new double[3]{ 0.89907336, 0.20947775, -0.78829497 } }, {"m_avg_L_Elbow", new double[3]{ 0.42310753, 0.3573367, -0.9505372 } }, {"m_avg_R_Elbow", new double[3]{ 0.9305408, 0.37685162, -0.62403321 } }, {"m_avg_L_Wrist", new double[3]{ 0.44996896, 0.49648461, -1.49405789 } }, {"m_avg_R_Wrist", new double[3]{ 0.83942693, 0.51275945, -0.99922866 } }, {"m_avg_Head", new double[3]{ 0.67040175, 0.1112375, -1.85335028 } }, }; As a result of applying points to parts of the SMPL model, I get an unpredictable result: I also tried to visualize the application of points to the SMPL model using the Smplx library in a python project, but the results are the same, the model assumes an incorrect pose. Most likely, I'm missing something, because when visualizing the skeleton through your code, it seems that it is correct...

Have you solved this problem？

[AndreAhmed]

@matlab2017 have you solved it ?