can't reproduce similar performance on waymo dataset

[ljyan93]

Background:

• config: tools/cfgs/waymo_models/voxelnext2d_ioubranch.yaml

• pytorch version: 1.9.1+cu111

• spconv version: 2.2.6

• loss log:
  Train: 12/12 (100%) [ 988/989 (100%)] Loss: 4.254 (5.10) LR: 3.015e-08 Time cost: 14:15/00:00 [2:56:45/00:00] Acc_iter 11868 Data time: 0.00(0.06) Forward time: 0.29(0.81) Batch time: 0.30(0.87)

I get zero AP/APH on all object types. And looks like I am getting a lot of weird extents

Tiny box dim seen, return 0.0 IOU.
b1: center_x: 5.7256546020507812
center_y: 11.504196166992188
center_z: 2.9886384010314941
width: 22.703926086425781
length: 560.6590576171875
height: 0.00077404256444424391
heading: -2.1097447872161865

could you hint a bit where I may do it wrong?

[yukang2017]

Hi,

This is really weird.

Would you please provide me your total log file and tensorboard file?
What number of GPUs you use?
Would you please tell me your e-mail? I will send your my checkpoint. You can have a test on it to check if the problem exist in training or testing.

Regards,
Yukang Chen

[ljyan93]

Discussed with Yukang, and found out it should be related to spconv version.

using sponv2.1 instead of spconv2.2 should fix the issue above.