Try to reproduce performance in paper

[Eaphan]

Thanks for sharing.
This is my reproduction results. The experiment is running on 8 gpus and config is not changed.

2022-01-10 21:31:31,266 INFO recall_roi_0.5: 0.964236
INFO:btcdet.utils.common_utils:recall_roi_0.5: 0.964236
2022-01-10 21:31:31,266 INFO recall_rcnn_0.5: 0.967569
INFO:btcdet.utils.common_utils:recall_rcnn_0.5: 0.967569
2022-01-10 21:31:31,266 INFO recall_roi_0.7: 0.816458
INFO:btcdet.utils.common_utils:recall_roi_0.7: 0.816458
2022-01-10 21:31:31,267 INFO recall_rcnn_0.7: 0.868125
INFO:btcdet.utils.common_utils:recall_rcnn_0.7: 0.868125
2022-01-10 21:31:31,267 INFO ***************************
INFO:btcdet.utils.common_utils: ***************************
2022-01-10 21:31:31,267 INFO precision: 0.531, recall: 0.717, f1: 0.599, precision_factored: 0.536 recall_factored: 0.723, f1_factored: 0.604
INFO:btcdet.utils.common_utils:precision: 0.531, recall: 0.717, f1: 0.599, precision_factored: 0.536 recall_factored: 0.723, f1_factored: 0.604
2022-01-10 21:31:31,267 INFO occ thresh 0.1: 0.953, occ thresh 0.2: 0.953, occ thresh 0.3: 0.953, occ thresh 0.4: 0.938, occ thresh 0.5: 0.914, occ thresh 0.6: 0.883, occ thresh 0.7: 0.833, occ thresh 0.8: 0.720, occ thresh 0.9: 0.233,
INFO:btcdet.utils.common_utils:occ thresh 0.1: 0.953, occ thresh 0.2: 0.953, occ thresh 0.3: 0.953, occ thresh 0.4: 0.938, occ thresh 0.5: 0.914, occ thresh 0.6: 0.883, occ thresh 0.7: 0.833, occ thresh 0.8: 0.720, occ thresh 0.9: 0.233,
2022-01-10 21:31:31,267 INFO total_pos_all_portion 0.599
INFO:btcdet.utils.common_utils: total_pos_all_portion 0.599
2022-01-10 21:31:31,269 INFO Average predicted number of objects(3769 samples): 4.237
INFO:btcdet.utils.common_utils:Average predicted number of objects(3769 samples): 4.237
pc, rc (1, 3, 2, 41) (1, 3, 2, 41)
pc, rc (1, 3, 2, 41) (1, 3, 2, 41)
2022-01-10 21:31:37,601 INFO Car AP@0.70, 0.70, 0.70:
bbox AP:90.6649, 89.7338, 89.2758
bev AP:90.0933, 88.1567, 87.6482
3d AP:89.0296, 85.5022, 78.6749
aos AP:44.18, 44.36, 44.10
Car AP_R40@0.70, 0.70, 0.70:
bbox AP:96.6044, 95.3285, 92.9333
bev AP:93.5117, 91.5184, 89.2229
3d AP:92.0635, 85.2752, 82.9497
aos AP:45.29, 43.36, 42.14
Car AP@0.70, 0.50, 0.50:
bbox AP:90.6649, 89.7338, 89.2758
bev AP:90.6736, 89.7183, 89.2712
3d AP:90.6736, 89.7061, 89.2479
aos AP:44.18, 44.36, 44.10
Car AP_R40@0.70, 0.50, 0.50:
bbox AP:96.6044, 95.3285, 92.9333
bev AP:96.6282, 95.4054, 92.9708
3d AP:96.6199, 95.3687, 92.9303
aos AP:45.29, 43.36, 42.14

[Eaphan]

Are there some other implementation details that we should pay attention to?

[Xharlie]

I have 4gpu and this is the code i use to produce the number in paper. Maybe you can take a look at issue #2

[Eaphan]

bbox AP:90.7997, 89.8695, 89.4887
bev AP:90.3523, 88.4955, 88.0498
3d AP:89.6627, 85.7698, 79.1362
aos AP:42.01, 40.39, 40.26
Car AP_R40@0.70, 0.70, 0.70:
bbox AP:96.7518, 95.4510, 93.1056
bev AP:95.6828, 91.7892, 89.5549
3d AP:92.8319, 85.8193, 83.5272
aos AP:44.75, 42.90, 41.91
Car AP@0.70, 0.50, 0.50:
bbox AP:90.7997, 89.8695, 89.4887
bev AP:90.7867, 89.8871, 89.5573
3d AP:90.7867, 89.8804, 89.5333
aos AP:42.01, 40.39, 40.26
Car AP_R40@0.70, 0.50, 0.50:
bbox AP:96.7518, 95.4510, 93.1056
bev AP:96.7762, 95.5894, 95.3303
3d AP:96.7687, 95.5493, 95.2496
aos AP:44.75, 42.90, 41.91

The bad performance resulted from my setting batch_size=1 wrongly. When I use right batch_size the performance is better and closer to claimed performance.