The different RR in test.py and eval.py

[Hui-design]

Hello, thank you for you amazing work again.
When I evaluated on 3DMatch, I found the RR reported by test.py and eval.py are very different, what makes this difference?

[35p32]

The definition of whether one pair of point cloud was successfully registered in test and eval stage is different. Check the code.

[qinzheng93]

RR in test.py is defined as mean error, while RR in eval.py is defined as RMSE following Redwood. Use the results from eval.py in your paper.

[Hui-design]

Thank you for your reply. I agree with that. Apart from the definition of RR, I also find that you remove consecutive frames (may following predator) in eval.py, which also makes the dataset more difficult, resulting in the reduction in RR from "test" to "eval". Is my understanding correct?

[qinzheng93]

Yes. The convention of skipping consecutive frames is from 3DMatch.

[sss752990616]

Hi @qinzheng93,

Thanks for your great work. As a beginner in this area, I just have one follow-up question, what is the main difference between test.py and eval.py, and why do we need both of them? It looks like they are doing similar things.

[sss752990616]

And one more question is that according to my reading of the code for eval.py and test.py, they look like both evaluate the same dataset. Not sure if my observation is correct.