Inquiry about the result with the uploaded pre-trained models

[lim0606]

Hi, this is Jaehyun Lim

First of all, thank you for sharing your work :)

I tried to run the posenets with the uploaded pretrained models by using the provided caffe version of yours.

However, the performance (median error) of the posenets with following settings and the pretrained models does not match with the reported performance in your paper.

• non-bayesian
• single center crop

I got

• 7.98 m and 4.36 deg (median error) for KingsCollege
• 3.45 m and 6.61 deg (median error) for ShopFacade

I further tried followings;

• converted images to lmdb with different interpolation settings, both bilinear and bicubic, since it was sometimes matter on some dataset; however, the results aren't changed much
  • 8.12 m and 4.30 deg (median error) for KingsCollege
  • 3.16 m and 6.28 deg (median error) for ShopFacade
• ran test with bayesian setting by using test_bayesian_posnet.py and adding DROPOUT layer at somewhere icp9 according to the provided test_bayesian_posenet.prototxt; however, it also provided almost the same results.
  • 8.12 m and 4.31 (median error) for KingsCollege

I would appreciate if you give me some advises about possible mistakes that I might make.

Sincerely,

Jaehyun

[alexgkendall]

Hi Jaehyun,

I've just looked into this and found there was a bug in the dataset creation script - sorry! I've just pushed an update to the code to fix this. Essentially there was a discrepancy between how scipy and opencv import image data. For my PoseNet work I used opencv, so I've changed the script to reflect this.

You will need to:

1. get the latest code
2. create new LMDB datasets using the new script
3. compute new dataset mean files
4. retest using these

Let me know how this works, you should be able to reproduce the figures in the ICCV paper.

Alex

[lim0606]

Hi, @alexgkendall

Thank you for your kind response!

I re-converted lmdb data set as you commented, and it works really well :)

For non-bayesian and single center crop inference, I got

• 1.922 m and 2.698 deg (median error) for KingsCollege
• 2.311 m and 2.695 deg for OldHospital
• 1.462 m and 4.039 deg for ShopFacade
• 2.651 m and 4.244 deg for StMarysChurch
• 3.129 m and 2.898 deg for Street

For bayesian inference (I'm not sure the uploaded weights were trained in this fashion, but I just ran it for fun)

• 1.613 m and 2.397 deg (median error) for KingsCollege

Once again, thank you for kind response!

Jaehyun

[alexgkendall]

Awesome!