Learning to deform mesh

[pieterblok]

I have a rather straightforward question: how can we properly learn to reconstruct a complete mesh / point cloud from a partial point cloud?

I'm inspired by the example deform_source_mesh_to_target_mesh. Yet as far as I understand there is nothing "learned" in this example: e.g. there is no neural network that captures the underlying deformation to reconstruct the complete / target point cloud (mesh).

Background for this question is the DeepSDF network that can learn to reconstruct a complete point cloud from a partial point cloud. I was able to make my own encoder-decoder approach using Pytorch3D's loss functions. The simple network compresses RGB-D frames from Intel Realsense camera into a 128 dimension latent vector (encoder) which is then used to reconstruct the complete 3D point cloud using the decoder. This is a approach similar to the one in #726.

Although I can visually inspect that my proposed encoder-decoder can partly reconstruct the target point cloud, it struggles of completing the actual shape: most points are clustered around the base of the target shape, without completing the far-right or far-left edges (e.g. density of predicted points is very high in the center of target shape and very low at the more extreme sides of the shape).

Currently, I'm using this very simple approach (pseudo code):

for e in range(param["epoch"]):
    for idx, item in enumerate(iter(dataset)):       
        optimizer.zero_grad()

        rgbd_image = torch.cat((item['rgb'], item['depth']), 1).to(device) ##dimension: [batch_size, 4]
        latent_vector = encoder(rgbd_image) ##dimension: [batch_size, latent_dim=128]
        new_verts = decoder(latent_vector) ##dimension: [batch_size, num_points=5000, point_dim=3]

        sample_trg = pytorch3d.io.load_objs_as_meshes(item['target_obj_files'], device=device) ##[batch_size, num_points=5000, point_dim=3]
        sample_pcd = pytorch3d.structures.Pointclouds(points=new_verts)
        loss_chamfer = pytorch3d.loss.point_mesh_edge_distance(sample_trg, sample_pcd)

        reg_loss = regularization_loss(new_verts)
        total_loss = loss_chamfer + param["lambda_reg"] * reg_loss

        total_loss.backward()
        optimizer.step()

    scheduler.step()

Is there a way to:

• convert the predicted points directly to a mesh in Pytorch3D and keep the network backward differentiable: this is similar to the deform_source_mesh_to_target_mesh example
• penalize points being predicted too close to each other (introduce a regularization method)
• loss functions based on mesh to mesh comparison (target vs prediction)

Thanks in advance!