Using NDF for assembly task

[freerafiki]

Hi @anthonysimeonov ,
Congrats for your work, I really appreciate the idea.

I was wondering whether the descriptors could be used for a different but related task. I am working on 3D assembly (see the breaking-bad dataset for a visual explanation), which means trying to assemble two (or more) broken parts of an object.
The idea behind using NDF is that grasping is similar to assembly, because we have two complementary part (the grasp from the robot or the broken part of the object). It is also equi-variant to rotation, which is great for assembling (if you rotate all pieces solution is still ok).

So I was playing around with the code to do some experiments, I can get the latent vector and forward it to get the descriptors, but I am unsure how exactly to use them. Instead of sampling one random 'batch' (many points close to each other outside the surface of the mesh) I am sampling around the whole object and creating the descriptors, and now I would want to match the descriptors from one broken object with the descriptors from the other part of the broken object.

Intuitively, I would expect the descriptors to be complementary (also not sure how to mathematically define complementary in this case), but I see in section II (Method) of the paper, Equation 11 says that minimizing the difference between descriptors is the way to get the transformation. But trying to find correspondences in a "standard" way (i was using example from TEASER++ registration) did not work.

In order to understand better, I was trying to investigate further the concept of energy landscapes, which looks very promising. Can you point me to some code (even part of this repo) to watch to get a better understanding of it?

So, a couple of questiosn:

• if I have a descriptor on a pointcloud (first broken part) and on a second pointcloud (second broken part, same object) and if they belong to the same point, should the descriptors be the same or the complementary?
• how did you manage to create the visualization of the energy landscapes? Where can I look to manage to get one of my own?
• do you think there is a way to use NDF for assembly?

Thanks a lot in advance