Matlab version

[MengXinChengXuYuan]

Hi thanks for the dataset and code
What's your matlab version? I can't run it under R2015a

Actually I writed the visual tool in python, using the smpl param and projection matrix provided. However the mesh I got isn't that accurate, as attched below.
I checked that the vertice I get matches the one provided in the reconstruction file and I didn't find any problem.

Here is my code:
'''
def proj_HUMBI():
opt = get_opt_debug()
smpl = SMPL(opt)

data_root = '/mnt/108-sdd/human_recon/data/humbi/body_mesh/'
subjects = os.listdir(data_root)

for subject in subjects:

    if subject != 'subject_136':

        subject_path = data_root + subject + '/body/'

        result_path = '/xxx/humbi/body_mesh/{}_rendered/'.format(subject)

        if not os.path.exists(result_path):
            os.mkdir(result_path)

        ext_lines = open(subject_path + 'extrinsic.txt', 'r').readlines()
        num_cam = int(ext_lines[1].replace('\n', '').split(' ')[-1])
        ext_lines = ext_lines[3:len(ext_lines)]

        int_lines = open(subject_path + 'intrinsic.txt', 'r').readlines()
        int_lines = int_lines[3:len(int_lines)]

        pro_lines = open(subject_path + 'project.txt', 'r').readlines()
        pro_lines = pro_lines[3:len(pro_lines)]

        cam_int_list = {}
        cam_ext_list = {}
        cam_pro_list = {}

        for i in range(num_cam):

            index_ext = ext_lines[5 * i].split(' ')[-1].replace('\n', '')

            cam_ext = np.zeros((4, 4)).astype(np.float32)

            for j in range(3):
                line_R = ext_lines[5 * i + 2 + j].replace('\n', '').split(' ')
                for k in range(3):
                    cam_ext[j][k] = float(line_R[k])

            line_T = ext_lines[5 * i + 1].replace('\n', '').split(' ')
            for j in range(3):
                cam_ext[j][3] = float(line_T[j])

            cam_ext[3][3] = 1
            cam_ext_list[index_ext] = cam_ext

            index_int = int_lines[4 * i].split(' ')[-1].replace('\n', '')

            cam_int = np.zeros((3, 3)).astype(np.float32)
            for j in range(3):
                line= int_lines[4 * i + 1 + j].replace('\n', '').split(' ')
                for k in range(3):
                    cam_int[j][k] = float(line[k])

            cam_int_list[index_int] = cam_int

            index_pro = pro_lines[4 * i].split(' ')[-1].replace('\n', '')
            project = np.zeros((3, 4)).astype(np.float32)
            for j in range(3):
                line= pro_lines[4 * i + 1 + j].replace('\n', '').split(' ')
                for k in range(4):
                    project[j][k] = float(line[k])

            cam_pro_list[index_pro] = project


        frames = os.listdir(subject_path)
        frames = [frame for frame in frames if frame.split('.')[-1] != 'txt']

        for frame in frames:

            print("processing {} {}".format(subject, frame))

            frame_path = subject_path + frame + '/'
            frame_image_path = frame_path + 'image/'
            frame_param_path = frame_path + 'reconstruction/smpl_parameter.txt'

            sub_result_path = result_path + frame + '/'

            if not os.path.exists(sub_result_path):
                os.mkdir(sub_result_path)

            smpl_param_lines = open(frame_param_path, 'r').readlines()

            scale = float(smpl_param_lines[0].replace('\n', ''))

            trans = []
            for i in range(1, 4):
                trans.append(float(smpl_param_lines[i].replace('\n', '')))
            trans = np.array(trans)

            pose = []
            for i in range(4, 76):
                pose.append(float(smpl_param_lines[i].replace('\n', '')))
            pose = np.array(pose)

            shape = []
            for i in range(76, 86):
                shape.append(float(smpl_param_lines[i].replace('\n', '')))
            shape = np.array(shape)

            # trans = np.insert(trans, 3, values=0)
            trans = trans.reshape(1, 3)

            pose = torch.from_numpy(pose.astype(np.float32)).unsqueeze(0)
            shape = torch.from_numpy(shape.astype(np.float32)).unsqueeze(0)

            vertice = smpl(pose, shape)  # * np.array(frame_data['scale']).astype(np.float32)
            pnts = smpl.get_joints(vertice).squeeze().detach().numpy()
            pnts = pnts * scale + trans
            pnts = np.insert(pnts, 3, values=1, axis=1)

            vertice = vertice.squeeze().detach().numpy()
            vertice = vertice * scale + trans
            vertice = np.insert(vertice, 3, values=1, axis=1)

            images = os.listdir(frame_image_path)
            for image_name in images:

                image = cv2.imread(frame_image_path + image_name)

                cam_index = str(int(image_name.replace('.jpg', '').replace('image', '')))

                cam_int = cam_int_list[cam_index]
                cam_int = np.insert(cam_int, 3, values=0, axis=1)

                cam_ext = cam_ext_list[cam_index]

                project = cam_pro_list[cam_index] # np.dot(cam_int, cam_ext)

                # pnt_temp = (pnts + trans).transpose(1, 0)
                # pnt24_3d_proj = np.dot(project, pnt_temp).transpose(1, 0)
                pnt24_3d_proj = np.dot(project, pnts.transpose(1, 0)).transpose(1, 0)
                vertice_3d_proj = np.dot(project, vertice.transpose(1, 0)).transpose(1, 0)

                for i in range(24):
                    pnt24_3d_proj[i][0] = pnt24_3d_proj[i][0] / pnt24_3d_proj[i][2]
                    pnt24_3d_proj[i][1] = pnt24_3d_proj[i][1] / pnt24_3d_proj[i][2]

                for i in range(6890):
                    vertice_3d_proj[i][0] = vertice_3d_proj[i][0] / vertice_3d_proj[i][2]
                    vertice_3d_proj[i][1] = vertice_3d_proj[i][1] / vertice_3d_proj[i][2]

                for i in range(24):
                    cv2.circle(image, (int(pnt24_3d_proj[i][0]), int(pnt24_3d_proj[i][1])), 5, (255, 0, 0))

                for i in range(6890):
                    cv2.circle(image, (int(vertice_3d_proj[i][0]), int(vertice_3d_proj[i][1])), 1, (0, 0, 255))

                cv2.imwrite(sub_result_path + '{}'.format(image_name), image)

In brief, read the projection matrix p, cal the mesh v using pose and shape, and p * (v * scale + trans)

And the result I can get:

Did i miss some thing, or the smpl param you provided is inaccurate in certain cases?

[MengXinChengXuYuan]

One more question, did you used only netural model for all subjects?
I noticed there are only a few frames provided (seems every 8 frames), do we have the rest frames or they just remain unopened?

[halcyon370]

Same problem with hand reconstruction.

[MengXinChengXuYuan]

@halcyon370 Hi what do you mean by same problem?
The matlab version or you also found that the reconstruction meshes are inaccurate in some cases ?

[halcyon370]

@MengXinChengXuYuan The reconstruction meshes of hand are also inaccurate. I use matlab of version 2018b.

[MengXinChengXuYuan]

@halcyon370 I see!

[zhixuany]

@MengXinChengXuYuan I think the visualization code itself is good. I will ping Jae Shin to try to address your issue since he was working on SMPL body mesh fitting. But as far as I know, we admit that it is not perfect.

[zhixuany]

@halcyon370 Similarly, I think the visualization code for hand itself is good. But as you are aware of, for some cases the fitting is not perfect and we will try to correct those cases in next round of refinement.

[quyanqiu]

@zhixuany Just curious about your hand fitting procedure,In your paper,you mentioned that you use 2D-3D reprojection error term to reconstruct the MANO hand model,But in your annotation, you provide us 3d keypoints GroundTruth,why you did not use the 3D-3D Euclidean distance error to fit mano, Is that more accurate than your current method?

[MengXinChengXuYuan]

@zhixuany Actually I think no one can generate perfect smpl params for now, in most cases :p
Thank you for sharing your data, and looking forward to the refinement!

[Gorokke]

@MengXinChengXuYuan Thanks for having interest in our dataset. The ground-truth that we can perfectly ensure is the "Multiview images". For mesh reconstruction, due to the fitting error (e.g., the error from the 3D keypoint reconstruction propagates to the mesh fitting), it could be inaccurate for some cases. - Jae Shin

[zhixuany]

@quyanqiu Actually I did "use the 3D-3D Euclidean distance error to fit mano" as you said. In "section 3.3. Hand" of paper we mentioned that "We align the hand model to multiview images by minimizing the Euclidean distance between hand keypoints". Is this sentence somehow misleading?

[quyanqiu]

@zhixuany sorry for misunderstanding,When I check the formular in your appendix its truly 3D-3D loss

[zhixuany]

@quyanqiu No problem. Thanks for being interested in HUMBI~