Merge request with OpenPCDet

[CSautier]

Hi,

I've been working on a merge of your codebase with the official OpenPCDet from OpenMMLab for the supervised part. The purpose would be that OpenPCDet would support supervised training on ONCE directly.

I would like to make a PR of my merge on OpenPCDet when it's ready, but I will wait for your agreement on that. Also if you have specific requirements on License, citation or parts of the code I should not include, don't hesitate to tell me, and we can see with OpenMMLab what can be done.

[PointsCoder]

@CSautier Sounds great. I really appreciate this great work.

Could you please share with me your PR before you want to merge to PCDet? I might want to make some modifications to the readme if possible. The main maintainer of PCDet is my previous labmate so it could be better if I talk to him ahead of time.

[CSautier]

I added you as a collaborator on the fork at https://github.com/CSautier/OpenPCDet/tree/ONCE
So far I have been able to reproduce results for SECOND with spconv 2, and am running PV-RCNN. I expect that reproducing results for CenterPoint will be the most difficult as the code in OpenPCDet changed a lot.

[PointsCoder]

@CSautier Sounds good. I made some modifications to the document and have no other concerns. Feel free to submit this PR to OpenPCDet, and let me know if there are any problems.

From previous feedback the results of CenterPoint are not quite stable, ranging from 58 to 60. It is mainly because the training set is quite small and different random seeds could lead to quite different results. I think it's ok and you can report your own reproduction results to OpenPCDet.

[CSautier]

Ok, thanks for the feedback. I have been able to reproduce similar results as in your article for PVRCNN, SECOND and PointRCNN. As I expected, CenterPoint seems to follow a different distribution, I added a warning in the README for now, and will suggest putting values of a new 8GPUS run later on. I am still running a model on PointPillars, and if it works normally, will put the PR to OpenPCDet in the next days.

[PointsCoder]

Hi,

I would suggest you report your own reproduced results here https://github.com/CSautier/OpenPCDet/tree/ONCE#once-3d-object-detection-baselines, which would be more reasonable as people who use OpenPCDet can easily reproduce the results.

To resolve the inconsistent performance of CenterPoint between reproduction and paper, we can add a shortcut to this repo for their reference and reproduction.

[PointsCoder]

Also, I'd like to know your reproduced results of CenterPoint. I noticed some config changes here https://github.com/CSautier/OpenPCDet/blob/812d2b8cac59accb88a1a0729f9f7857db3c70b5/tools/cfgs/once_models/centerpoint.yaml#L70. The loc_weight I set in this repo should be 0.25. Will that influence the performance?

[CSautier]

Hi,
I think except for CenterPoint, there is no need to report new results, since the code is almost identical, but doing so would make that results in semi-supervised and self-supervised settings would no longer be comparable.
For CenterPoint, I only did a single GPU run, which ended with an mAP of 65.47, more than 5 points above your baseline. I would need to report an 8 GPU run in the table instead.
As for the config, when it was not immediately clear how they should transpose, I used the config of waymo or default values instead. I will try a run with loc_weight =0.25

[PointsCoder]

I am surprised you get a much higher performance with one GPU training. That might be because more training iterations lead to better performance.

We should remove the words "*Results are not reproducible with this version of the code" if you didn't try 8-GPU training. If you tried that you can update your own result, which would be clearer.

[CSautier]

Actually, my experiments on both SECOND and PVRCNN show an improvement with a single GPU training of about 2 mAP points.
I agree with your remark for CenterPoint, as soon as I have an 8-GPU training for it, I will put the values in the table and remove the warning.

[CSautier]

I finished two training on 8 GPUs with the new code of centerpoint, with different values of loc_weight.
Overall the score improved a bit, I will report those new scores in the README, and be in touch with OpenPCDet to update those on their repo as well.

loc_weight = 1.0
|AP@50 |overall |0-30m |30-50m |50m-inf |
|Vehicle |78.02 |88.32 |72.35 |60.11 |
|Pedestrian |49.74 |57.83 |40.58 |23.45 |
|Cyclist |67.22 |77.50 |61.91 |45.68 |
|mAP |64.99 |74.55 |58.28 |43.08 |

loc_weight = 0.25
|AP@50 |overall |0-30m |30-50m |50m-inf |
|Vehicle |75.43 |86.11 |69.67 |54.96 |
|Pedestrian |48.21 |55.12 |41.01 |26.48 |
|Cyclist |65.43 |75.80 |59.55 |43.36 |
|mAP |63.02 |72.34 |56.74 |41.60 |