SHIFT_COOR？

[AndyYuan96]

Hi，jihan, the parameter SHIFT_COOR in cfg file pvrcnn_old_anchor_ros.yaml means what? does SHIFT_COOR is the lidar position difference among different dataset?

[jihanyang]

Yes. Different datasets have different denfination of original point. I shift the point clouds to match the defination in waymo: the original point is put on the road plane. This parameter is set acoording to observation.

[AndyYuan96]

Yes. Different datasets have different denfination of original point. I shift the point clouds to match the defination in waymo: the original point is put on the road plane. This parameter is set acoording to observation.

I use the config pvrcnn_old_anchor_ros.yaml, and I can't reproduce the paper result。I use 1/5 trainset of waymo dataset to pretrain as I can train within oneday with 1/5 trainset， and direct test checkpoint_30.pth on kitti dataset。
Result：
INFO Car AP@0.70, 0.70, 0.70
box AP:85.9941, 81.6353, 80.6863
bev AP:67.2011, 60.1171, 58.6911
3d AP:15.4357, 14.6012, 14.8312
aos AP:85.93, 81.35, 80.34

but the iou=0.5's result is relative normal：
Car AP@0.70, 0.50, 0.50
box AP:85.9941, 81.6353, 80.6863
bev AP:94.4203, 90.3825, 87.1545
3d AP:87.7402, 85.8818, 86.2095
aos AP:85.93, 81.35, 80.34

do you have any advise for me to find the reason？does 1/5 and 1/2 trainset have so much different on result?

[AndyYuan96]

what's more, I use waymodataset1.2, and I use toplidar and four blind(补盲) lidar。

[jihanyang]

Hello,
Can you eval all epochs of the results and provide the best result? I am not sure the influence of 1/5 and 1/2, I guess it should not be so influential. Also, I have discussed waymo dataset version with shaoshuai, and we think this should not be very influential. Notice that the detector fluctuates epoch by epoch when the target domain is kitti, so we always select the best result along all epochs.

[AndyYuan96]

Hello,
Can you eval all epochs of the results and provide the best result? I am not sure the influence of 1/5 and 1/2, I guess it should not be so influential. Also, I have discussed waymo dataset version with shaoshuai, and we think this should not be very influential. Notice that the detector fluctuates epoch by epoch when the target domain is kitti, so we always select the best result along all epochs.

OK，I will try

[AndyYuan96]

Hello,
Can you eval all epochs of the results and provide the best result? I am not sure the influence of 1/5 and 1/2, I guess it should not be so influential. Also, I have discussed waymo dataset version with shaoshuai, and we think this should not be very influential. Notice that the detector fluctuates epoch by epoch when the target domain is kitti, so we always select the best result along all epochs.

it looks like that the cfg file pvrcnn_old_anchor_sn.yaml and pvrcnn_old_anchor_ros.yaml are the same......😭

[AndyYuan96]

I use cfg pvrcnn_old_anchor_ros.yaml to train, with batch size 40(8gpu)，and test use cfg pvrcnn_st3d.yaml。after evaluation the checkpoint_[16,30], total 15 checkpoints, the best result I get for 3d ap(iou=0.7) is 25, I'm now evaluation the before checkpoint, checkpoint_[0, 15]

[AndyYuan96]

unfortunately, the checkpoint_[0,15]'s result is more low, so the total 30 checkpoints's best result for 3d AP(iou=0.7) is 25.

[jihanyang]

Hello,
Can you eval all epochs of the results and provide the best result? I am not sure the influence of 1/5 and 1/2, I guess it should not be so influential. Also, I have discussed waymo dataset version with shaoshuai, and we think this should not be very influential. Notice that the detector fluctuates epoch by epoch when the target domain is kitti, so we always select the best result along all epochs.

it looks like that the cfg file pvrcnn_old_anchor_sn.yaml and pvrcnn_old_anchor_ros.yaml are the same......😭

I am quick sorry for my mistake. I will fix this problem quickly. I think we can test the testing code first. You can send me an email, and I will send my pretrained model to you (due to waymo license, I cannot make it public). If the problem does not lies in the testing code, I will then try to use 1/5 data and v1.2 data. Don't worry about that~

[AndyYuan96]

Hello,
Can you eval all epochs of the results and provide the best result? I am not sure the influence of 1/5 and 1/2, I guess it should not be so influential. Also, I have discussed waymo dataset version with shaoshuai, and we think this should not be very influential. Notice that the detector fluctuates epoch by epoch when the target domain is kitti, so we always select the best result along all epochs.

it looks like that the cfg file pvrcnn_old_anchor_sn.yaml and pvrcnn_old_anchor_ros.yaml are the same......😭

I am quick sorry for my mistake. I will fix this problem quickly. I think we can test the testing code first. You can send me an email, and I will send my pretrained model to you (due to waymo license, I cannot make it public). If the problem does not lies in the testing code, I will then try to use 1/5 data and v1.2 data. Don't worry about that~

already send you a email to paper email address。

[jhzhang19]

@AndyYuan96 Hello, what's the result of your reproduction? Is it close to the effect in the paper?

[AndyYuan96]

@AndyYuan96 Hello, what's the result of your reproduction? Is it close to the effect in the paper?

yes, now I can reproduce the result, some notes:

1. use spconv1.0 and openpcdet0.2 rather than spconv1.2 and openpcdet0.3, for spconv1.2 and openpcdet0.3, I can't reproduce.
2. for pertained model, chooses the best ap model using kitti val dataset to eval.

[jhzhang19]

@AndyYuan96 Hello, what's the result of your reproduction? Is it close to the effect in the paper?

yes, now I can reproduce the result, some notes:

1. use spconv1.0 and openpcdet0.2 rather than spconv1.2 and openpcdet0.3, for spconv1.2 and openpcdet0.3, I can't reproduce.
2. for pertained model, chooses the best ap model using kitti val dataset to eval.

Thank you for your advice, i will try it.

[chaytonmin]

Could you share your results and CONFIGS as you reproduce the result. I have tried many times, but the performance was still far away from the paper

[Galaxy-ZRX]

Hello @AndyYuan96 , I am also trying to reproduce this work and met some issues. I use spconv 1.0 and pcdet 0.2, I can achieve 65% on waymo for pre-trained step, then I go to the self-train step. However, after that, I can only get this result:

bbox AP:86.0902, 76.7667, 76.7454
bev AP:44.3801, 41.8471, 38.3252
3d AP:3.4005, 4.7295, 4.7189
aos AP:86.05, 76.54, 76.44
Car AP_R40@0.70, 0.70, 0.70:
bbox AP:89.0660, 80.5046, 80.4434
bev AP:40.1157, 38.4748, 35.1805
3d AP:2.9459, 3.8032, 3.6749
aos AP:89.02, 80.22, 80.04
Car AP@0.70, 0.50, 0.50:
bbox AP:86.0902, 76.7667, 76.7454
bev AP:96.9185, 87.5368, 87.5607
3d AP:88.8829, 86.0001, 85.9327
aos AP:86.05, 76.54, 76.44
Car AP_R40@0.70, 0.50, 0.50:
bbox AP:89.0660, 80.5046, 80.4434
bev AP:97.7307, 90.5892, 90.7585
3d AP:94.1681, 87.2719, 87.3523
aos AP:89.02, 80.22, 80.04

I dont know why the AP for threshold 70 and 50 can have such a large gap. Could you give me some help? My email address is 18916835036@163.com or you can just reply here (depends on your convenience!), thank you very much!