Can this work with purely synthetic training data (and no texture map)?

[murrdpirate]

I like how this approach does not require a texture of the object's 3D model. As you mention in the paper, this is a typical situation, especially for objects that are challenging to scan.

In some cases, there are also no real images of the object to train on. In these cases, synthetic data with domain randomization is often used to render images with random textures applied to the object's 3D model.

If I understand correctly, RePOSE trains on real images of the object. Have you done any experimentation with just domain-randomized synthetic data? Any intuition on whether this will work?

[sh8]

Hi @murrdpirate! Thank you so much for your interest in our project!

In some cases, there are also no real images of the object to train on. In these cases, synthetic data with domain randomization is often used to render images with random textures applied to the object's 3D model.

The important thing is that creating synthetic images still requires a textured 3D model. RePOSE can perform decently well even trained on only real images (no texture map is needed at all) but we needed to use some synthetic images for a fair comparison with other methods.

If I understand correctly, RePOSE trains on real images of the object. Have you done any experimentation with just domain-randomized synthetic data? Any intuition on whether this will work?

As mentioned above, it is not ideal but we use both synthetic and real images. I think RePOSE should work if you synthesize images with a high-quality textured 3D model, but as you said it will lead to another challenge for RGB scanning. We actually propose an efficient way to collect large-scale real images for 3D object detection in another paper. It would be helpful if you want to tackle several challenging object categories hard to obtain an RGB texture.

[murrdpirate]

Thanks for the reply @sh8!

The important thing is that creating synthesis images still requires a textured 3D model.

What I'm wondering is: what if I did not have a texture for the model, and so I just applied random textures when rendering synthetic images? Would that have any hope in working with this procedure?

It's definitely a harder problem when no images with real texture are available, but I think it's still solvable.

[sh8]

Since RePOSE heavily relies on a visual feature from an observed image, I honestly think it's not that easy to make it work.
I like your idea and the settings of no images with real texture are realistic, so it should be an interesting future research direction!

[murrdpirate]

Appreciate the feedback!

[sh8]

No problem! Please ask me any questions anytime!