Prediction score threshhold stability

[hailanyi]

Hi!
Thanks for sharing the code. With out any code changes, I only got 79 AP11 on val set as below. It is weird, because the paper can go up to 86 AP11. Is there a trained model, or could you give me some advice for training @Xharlie ?

Car AP@0.70, 0.70, 0.70:
bbox AP:90.7842, 89.7048, 89.3668
bev AP:90.3260, 88.2499, 88.1178
3d AP:89.5578, 79.5970, 78.9717
aos AP:42.11, 40.24, 40.12
Car AP_R40@0.70, 0.70, 0.70:
bbox AP:96.9146, 93.4603, 90.9702
bev AP:93.9134, 89.9003, 87.4430
3d AP:92.9308, 84.1075, 81.5648
aos AP:44.93, 41.93, 40.84

[hailanyi]

I changed the score threshold (0.78->0.6) and now it reaches 85 AP11 as below

bbox AP:97.9524, 89.7096, 89.3596
bev AP:90.3292, 88.2457, 87.8269
3d AP:89.5458, 85.6965, 78.9677
aos AP:45.39, 40.24, 40.11
Car AP_R40@0.70, 0.70, 0.70:
bbox AP:98.7796, 95.2468, 92.9688
bev AP:95.6869, 91.5468, 89.3502
3d AP:92.7121, 85.5772, 83.2944
aos AP:45.77, 42.73, 41.75

[Xharlie]

Thanks for letting me know.
Looks like it's not very stable, since on my server, 78 gave me the best.

[Yzichen]

I got similar results with hailanyi, I don’t know if some of the given config parameters are not optimal, such as COORD_TYPE. Besides, I want to know the details of the model implementation, but there are many functions in occ_target that I don’t understand. If you have time, can you provide some code comments? thanks！

[Eaphan]

I changed the score threshold (0.78->0.6) and now it reaches 85 AP11 as below

bbox AP:97.9524, 89.7096, 89.3596 bev AP:90.3292, 88.2457, 87.8269 3d AP:89.5458, 85.6965, 78.9677 aos AP:45.39, 40.24, 40.11 Car AP_R40@0.70, 0.70, 0.70: bbox AP:98.7796, 95.2468, 92.9688 bev AP:95.6869, 91.5468, 89.3502 3d AP:92.7121, 85.5772, 83.2944 aos AP:45.77, 42.73, 41.75

I also try to reproduce the performance claimed in paper, but it's not as good as claimed number.
Could you get stable and high performance in each experiment? If not, how do you achieve very good performance on the test test? Look forward to your reply. Thanks! @Xharlie

[syy-whu]

I changed the score threshold (0.78->0.6) and now it reaches 85 AP11 as below
bbox AP:97.9524, 89.7096, 89.3596 bev AP:90.3292, 88.2457, 87.8269 3d AP:89.5458, 85.6965, 78.9677 aos AP:45.39, 40.24, 40.11 Car AP_R40@0.70, 0.70, 0.70: bbox AP:98.7796, 95.2468, 92.9688 bev AP:95.6869, 91.5468, 89.3502 3d AP:92.7121, 85.5772, 83.2944 aos AP:45.77, 42.73, 41.75

I also try to reproduce the performance claimed in paper, but it's not as good as claimed number. Could you get stable and high performance in each experiment? If not, how do you achieve very good performance on the test test? Look forward to your reply. Thanks! @Xharlie

Hello, do you reproduce the performance claimed in paper? I cannot reproduce such performance in the val set and test set

[Xharlie]

Hi I use this code base to produce the results, however seems the threshold on my machine may not be the same for everyone. I also observe the number of gpus used in training will somehow impact the final results, not sure if it causes the problem

[syy-whu]

Hi I use this codebase to produce the results, however, seems the threshold on my machine may not be the same for everyone. I also observe the number of gpus used in training will somehow impact the final results, not sure if it causes the problem.

Thanks for your reply, I will try four GPUs. By the way, when submitting the test results, how many epoches do you adopt? 40?