Projection of estimated SMPL/SMPL-X mesh on Image

[tommycwh]

Thank you very much for your amazing work. I would like to ask a question about how to project a mesh estimated by PyMAF/PyMAF-X back on the input image.

It seems that PyMAF projects the predicted joints using the projection(...) function and the predicted camera parameters as in here. I guess this function is also applicable to a set of vertices too, instead of just joints.

Implementation of projection(...) in smpl branch:

PyMAF/utils/geometry.py

Lines 241 to 255 in fb21d10

	def projection(pred_joints, pred_camera, retain_z=False):
	pred_cam_t = torch.stack([pred_camera[:, 1],
	pred_camera[:, 2],
	2 * 5000. / (224. * pred_camera[:, 0] + 1e-9)], dim=-1)
	batch_size = pred_joints.shape[0]
	camera_center = torch.zeros(batch_size, 2)
	pred_keypoints_2d = perspective_projection(pred_joints,
	rotation=torch.eye(3).unsqueeze(0).expand(batch_size, -1, -1).to(pred_joints.device),
	translation=pred_cam_t,
	focal_length=5000.,
	camera_center=camera_center,
	retain_z=retain_z)
	# Normalize keypoints to [-1,1]
	pred_keypoints_2d = pred_keypoints_2d / (224. / 2.)
	return pred_keypoints_2d

I want to project a SMPL-X mesh estimated by PyMAF-X on the input image too. However, I do not see the projection(...) function being used in the demo_smplx.py script. Also, I found the implementation of the projection(...) function somewhat different from its counterpart in the smpl branch.

Excerpt of projection(...) in smplx branch:

PyMAF/utils/geometry.py

Lines 379 to 401 in 18a68d8

	def projection(pred_joints, pred_camera, retain_z=False, iwp_mode=True):
	""" Project 3D points on the image plane based on the given camera info,
	Identity rotation and Weak Perspective (IWP) camera is used when iwp_mode=True, more about camera settings:
	SPEC: Seeing People in the Wild with an Estimated Camera, ICCV 2021
	"""
	batch_size = pred_joints.shape[0]
	if iwp_mode:
	cam_sxy = pred_camera['cam_sxy']
	pred_cam_t = torch.stack([cam_sxy[:, 1],
	cam_sxy[:, 2],
	2 * 5000. / (224. * cam_sxy[:, 0] + 1e-9)], dim=-1)
	camera_center = torch.zeros(batch_size, 2)
	pred_keypoints_2d = perspective_projection(pred_joints,
	rotation=torch.eye(3).unsqueeze(0).expand(batch_size, -1, -1).to(pred_joints.device),
	translation=pred_cam_t,
	focal_length=5000.,
	camera_center=camera_center,
	retain_z=retain_z)
	# # Normalize keypoints to [-1,1]
	# pred_keypoints_2d = pred_keypoints_2d / (224. / 2.)
	else:

Can I just use this function in this way, projection(dict(cam_sxy=pred_joints), pred_cam), to convert the predictions from PyMAF-X to the input of projection(...)? Or how should I use this function to get a correct projection?

Here is an example in which I project the estimated smplx vertices by: projection(dict(cam_sxy=smplx_verts), pred_cam). smplx_verts has shape (1,10475,3) and pred_cam has shape (1,3). The dimensions of the input image are (512,512). The (x and y) coordinates of the projected vertices range from -106.73 to 98.63. I am guessing that the domains of coordinates are like [-W/2, +W/2] and [-H/2, +H/2], so I rescale the projects coordinates by x*2+256 and y*2+256. Here are the input image (left) and the projection vertices (right) I get.

Am I doing the projection correctly? Although the projection somewhat aligns with the actual human shape on the image, I am not sure if I am just putting the projected points at this location by luck or not. Also, the projection is a little bit smaller than the actual shape, and I am not sure if this is just because it is an estimation using SMPL-X, or if I am not projecting the shape correctly.

Thank you!

[HongwenZhang]

Hi, thanks for your interest in PyMAF-X. projection(dict(cam_sxy=smplx_verts), pred_cam) seems to be not correct. Do you mean projection(smplx_verts, dict(cam_sxy=pred_cam))? Moreover, the input resolution of PyMAF-X should be 224x224.

[tommycwh]

Yes, I put the dictionary in the wrong argument while I was writing here. It should be projection(smplx_verts, dict(cam_sxy=pred_cam)).

About the resolution, I did not notice about the resolution used in PyMAF-X and I was just considering the resolution of my own images. I adjusted my way to visualize the projected points to (x+112)*(512/224) and (y+112)*(512/224), so that I first (1) change the projected vertices from [-112, +112] (as I guess) to [0,224], and then (2) rescale the points by 512/224 to match with the resolution of my own image. However, the projected points (left) I get is still different from the smplx mesh (right) visualized using the renderer in smplx_demo.py.

May I ask if I am correct about the domain of the projected vertices being [-112, +112]? Meanwhile, may I ask if there would be a recommended way to do the projection without using the renderer in smplx_demo.py script?

Thank you very much for your help.

[HongwenZhang]

The demo code of PyMAF/PyMAF-X first detects the person in the image, then crops a new image based on the detected person bounding box. So, I think the reprojected points should not be simply rescaled to the original image.

[tommycwh]

If the projection(...) function project 3D points to [-112,+112]^2 (for projected points that are inside the 224^2 bbox), is it possible to (1) first rescale the projected points from the resized bbox [-112,+112] to the original bbox [W_bbox,H_bbox], and then (2) translate the projected points on the original image according to the location of the original bbox? For example:

x' = (bbox_x0 - W_bbox/2) + (x + 112)*(W_bbox/224)
y' = (bbox_y0 - H_bbox/2) + (y + 112)*(H_bbox/224)

where

x, y: coordinates of the projected points (outputs of projection(...) function, on the resized bbox [-112,+112]^2)
x', y': "reprojected points" on the original image
W_bbox, H_bbox: dimensions of the original bbox

This (left) is what I get with this conversion, which does appear closer to the one rendered by smplx_demo.py (in my previous reply):

Do you think this procedure is doing the correct projection?

Besides, I am wondering if I am making things complicated here. Would there be parameters that already captures the bridges the resized bbox and the original image, which might make this "reprojection" easier?

[tommycwh]

I found the parameters to re-create the projection without using the renderer provided in the code (only when IWP camera mode is not used).

Following "SPEC: Seeing People in the Wild with an Estimated Camera", PyMAF(-X) seems to model the camera translation R_c in the world space, while assuming zero translation of the SMPL(-X) mesh and identity rotation of the camera.

Translating to the variables used in ````demo_smplx.py```, the parameters of the perspective camera are:

# Camera rotation
R = [[1., 0., 0.],
        [0., 1., 0.],
        [0., 0., 1.],]
# Camera translation
T = pred_results['orig_cam_t']
# Camera intrinsics
K = [[5000.,        0., img_w//2],
        [       0., 5000., img_h//2],
        [       0.,        0.,             1.],]

With these parameters, I can re-create the projection with other rendering libraries, e.g., PyTorch3D. I am doing this because there are other values about the estimated SMPL(-X) models I would like to render, and finding out the camera parameters is needed. The parameters might need to be adjusted to adapt to the underlying camera spaces if even other rendering libraries are used. Hope this will help others who are working on some similar problems.

Last but not least, thank you very much for the amazing work @HongwenZhang !

[HongwenZhang]

Yeah, the reprejection result is now correct. Glad to hear that you have figured it out! @tommycwh