About the intermediate results

[PanJiangSCU]

Hello author! Glad to find such an interesting project. Of course, I'm currently doing similar research, but I'm just getting started.

At present, I can complete the whole process with the mouse custom data you provided and the result is expected. However, the performance of the data I shot myself was not very satisfying.

The reference images that match the query image are transposed, like this.

But the second half of the test video is as expected. Why?
result video like this.
https://user-images.githubusercontent.com/114153764/195965944-60b5bdf2-ed43-4f88-ac75-edbe353fa665.mp4

Looking forward to your reply！

[liuyuan-pal]

Hi, thanks for your interest in our work. To improve the quality, I suggest

1. You may try to flip your z+ direction in your meta info. Because all your reference images are upside down, flipping the z+ direction will correct this.
2. You may resize the query image smaller because the object is too large and the scale difference between the query image and reference image is too large for the detector to find a correct scale.

[PanJiangSCU]

Thank you very much for your patient answer. I've had success with my own data.

[PanJiangSCU]

Hi! I now have some new questions.

1. Is there any way to evaluate the accuracy of Gen6D's estimatior on my own custom dataset?
2. I noticed that the viewpoint selector needs to select several reference images with close viewpoint to the query image. But the intermediate result shows that the reference images selected by the delector and the query image are quite different. like this.
   
3. I think there are reference images as follows that are closer to the viewpoint of the query image.
   
   Does this affect the result accuracy？
4. In sfm, how can I shoot around the object to make the experiment work better? Do point cloud reconstruction accuracy and the number of reference images used for reconstruction have a large impact on pose estimation?
5. How suitable is Gen6D for transparent objects?

[liuyuan-pal]

1. (1) In my experiments, I use COLMAP first to reconstruct object poses in two sequences separately and then manually align them by marking several correspondences on the object. In this case, we assume the aligned object poses estimated by COLMAP are ground-truth poses, which can be used for evaluation.
2. (2-3)The viewpoint selector may still make mistakes because all images look very similar. However, these mistakes are acceptable because they can be corrected by the subsequent refiner.
3. (4) The reconstruction quality of the point cloud is not important but the number of reference images is important. You may hold the cellphone to capture a video around the object and cover all viewpoints as possible.
4. (5) I'm also astonished that Gen6D also works for these transparent objects. Because Gen6D is based on matching images for pose estimation, there should be enough information like shape/contour/texture, etc. so that Gen6D is able to work. It would be an interesting research topic to improve the quality of object pose estimation on transparent objects!

[PanJiangSCU]

First of all, thank you very much for your patience. I wonder if sfm uses GPU for the whole run time? Running on my computer, it seems that my the GPU's utilization is always low.
I confirmed that my computer has installed the cuda pytorch1.12.0 with cuda11.6, and the python version is 3.9.13, the colmap version is colmap3.7, and my operating system is win10 and win11. The following picture shows the process of sfm on my device with Win10 OS.

[liuyuan-pal]

COLMAP is running the patch matching for dense reconstruction in your captured image. It is supposed to use GPUs but I'm not sure why it does not utilize GPUs. You may check nvidia-smi which would be more accurate to show GPU utilization.

[PanJiangSCU]

Thank you very much!