Unable to reproduce CBGS's results on NuScenes

[peiyunh]

Comparing to the current master branch, I made two changes in order to fix the NaN training loss.

The first change is described in #46 .

The second change is to add what's below before line 193 in losses.py

# FIX NaN TARGETS 
target_tensor = torch.where(
    torch.isnan(target_tensor), prediction_tensor, target_tensor
)

Besides, I set

norm_cfg = dict(type='SyncBN', eps=1e-3, momentum=0.01)

in examples/cbgs/configs/nusc_all_vfev3_spmiddleresnetfhd_rpn2_mghead_syncbn.py and

torch.backends.cudnn.benchmark = True

in tools/train.py.

Here are my results on the validation set after training 20 epochs:

car Nusc dist AP@0.5, 1.0, 2.0, 4.0
59.25, 71.87, 77.22, 79.63 mean AP: 0.7199402759604012
truck Nusc dist AP@0.5, 1.0, 2.0, 4.0
17.96, 35.01, 43.00, 47.15 mean AP: 0.357782470829584
construction_vehicle Nusc dist AP@0.5, 1.0, 2.0, 4.0
0.00, 1.28, 6.75, 13.37 mean AP: 0.05348830261303094
bus Nusc dist AP@0.5, 1.0, 2.0, 4.0
23.87, 48.49, 62.98, 66.32 mean AP: 0.5041451034213309
trailer Nusc dist AP@0.5, 1.0, 2.0, 4.0
1.94, 14.27, 30.88, 42.11 mean AP: 0.22300031478924093
barrier Nusc dist AP@0.5, 1.0, 2.0, 4.0
28.06, 48.97, 57.80, 60.27 mean AP: 0.4877375663669212
motorcycle Nusc dist AP@0.5, 1.0, 2.0, 4.0
24.97, 29.29, 30.38, 30.99 mean AP: 0.28906646690838084
bicycle Nusc dist AP@0.5, 1.0, 2.0, 4.0
6.20, 7.36, 7.98, 8.53 mean AP: 0.07516058303100348
pedestrian Nusc dist AP@0.5, 1.0, 2.0, 4.0
62.82, 64.73, 66.83, 69.03 mean AP: 0.658543997130018
traffic_cone Nusc dist AP@0.5, 1.0, 2.0, 4.0
42.10, 44.31, 46.23, 50.65 mean AP: 0.4582346501948114

Overall the mean AP is 38.2, which is much lower than what's reported.

Can someone point me to what I might have missed? Thanks!

[peiyunh]

Closing the issue as I found my ground truth velocities are incorrectly set (#46).

[muzi2045]

the same result, I have check the box velocity, it's normal

mAP: 0.3719
mATE: 0.3724
mASE: 0.2661
mAOE: 0.9296
mAVE: 1.3655
mAAE: 0.2684
NDS: 0.4023
Eval time: 140.1s

Per-class results:
Object Class	AP	ATE	ASE	AOE	AVE	AAE
car	0.721	0.219	0.158	0.841	1.116	0.230
truck	0.371	0.426	0.198	0.640	1.155	0.307
bus	0.500	0.439	0.174	1.223	2.171	0.431
trailer	0.213	0.687	0.219	0.670	1.371	0.184
construction_vehicle	0.058	0.798	0.481	1.370	0.157	0.372
pedestrian	0.653	0.165	0.287	1.350	0.869	0.439
motorcycle	0.242	0.223	0.243	1.107	3.192	0.153
bicycle	0.043	0.199	0.264	1.111	0.894	0.031
traffic_cone	0.449	0.170	0.348	nan	nan	nan
barrier	0.470	0.398	0.289	0.056	nan	nan
Evaluation nusc: Nusc v1.0-trainval Evaluation
car Nusc dist AP@0.5, 1.0, 2.0, 4.0
59.48, 71.97, 77.40, 79.65 mean AP: 0.7212472062431424
truck Nusc dist AP@0.5, 1.0, 2.0, 4.0
18.48, 36.29, 44.83, 48.91 mean AP: 0.3712787143771077
construction_vehicle Nusc dist AP@0.5, 1.0, 2.0, 4.0
0.00, 2.09, 8.06, 12.96 mean AP: 0.05777510817362395
bus Nusc dist AP@0.5, 1.0, 2.0, 4.0
23.60, 47.30, 62.94, 66.32 mean AP: 0.5003920838518946
trailer Nusc dist AP@0.5, 1.0, 2.0, 4.0
1.66, 13.24, 29.62, 40.49 mean AP: 0.21251682647224052
barrier Nusc dist AP@0.5, 1.0, 2.0, 4.0
26.14, 46.78, 55.95, 59.13 mean AP: 0.4700045657239055
motorcycle Nusc dist AP@0.5, 1.0, 2.0, 4.0
20.39, 24.74, 25.62, 26.05 mean AP: 0.24202605811125658
bicycle Nusc dist AP@0.5, 1.0, 2.0, 4.0
3.93, 4.27, 4.35, 4.58 mean AP: 0.04280152387228541
pedestrian Nusc dist AP@0.5, 1.0, 2.0, 4.0
62.12, 64.40, 66.13, 68.36 mean AP: 0.6525328516852104
traffic_cone Nusc dist AP@0.5, 1.0, 2.0, 4.0
40.81, 43.40, 45.49, 49.80 mean AP: 0.44874109465427564

[peiyunh]

I am waiting for my experiments with fixed ground truth velocities to finish, but so far it does not look like numbers are getting better than the ones that I posted here.

According to README.md, CBGS on NuScenes is not yet released officially. I am hoping the authors would address this issue.

Also, I am not sure why the velocity-related error (AVE) seems unreasonably large.

[muzi2045]

the boundingbox velocity x,y loss are abnormal large than other elements in the box.
it's really strange. are the velocity of obstacle is global velocity or ego velocity?
@peiyunh

[muzi2045]

OK, the CBGS use the global velocity of the annotations.
But I am using the ego velocity (in Lidar coords)

# convert velo from global to lidar
for i in range(len(ref_boxes)):
     velo = np.array([*velocity[i], 0.0])
     velo = velo @ np.linalg.inv(e2g_r_mat).T @ np.linalg.inv(
            l2e_r_mat).T
     velocity[i] = velo[:2]
velocity = velocity.reshape(-1,2)

maybe need to check velocity loss compute function....

[peiyunh]

Below are the results of my second run after fixing the ground truth velocities. What I did was to use the nuscenes devkit provided by the authors instead of the official devkit. Looks like numbers still do not match what's in the paper. Something must be off.

Saving metrics to: /home/peiyunh/work/Det3D/models/NUSC_CBGS_reproduce_20200107_230625 
mAP: 0.3775                                                                            
mATE: 0.3928                                                                           
mASE: 0.2657                                                                           
mAOE: 0.8910                                                                           
mAVE: 1.2791                                                                           
mAAE: 0.2503                                                                           
NDS: 0.4087                                                                            
Eval time: 79.5s                                                                       
2020-01-09 08:03:42,035 - INFO -                                                       
                                                                                       
2020-01-09 08:03:42,035 - INFO - Evaluation nusc: Nusc v1.0-trainval Evaluation        
car Nusc dist AP@0.5, 1.0, 2.0, 4.0                                                    
60.02, 71.89, 77.26, 79.49 mean AP: 0.7216487307140759                                 
truck Nusc dist AP@0.5, 1.0, 2.0, 4.0                                                  
17.64, 35.45, 44.55, 49.49 mean AP: 0.36782050865043414                                
construction_vehicle Nusc dist AP@0.5, 1.0, 2.0, 4.0                                   
0.00, 1.01, 5.64, 11.60 mean AP: 0.04562247557957591                                   
bus Nusc dist AP@0.5, 1.0, 2.0, 4.0                                                    
21.21, 45.50, 62.81, 66.61 mean AP: 0.4903582763408201                                 
trailer Nusc dist AP@0.5, 1.0, 2.0, 4.0                                                
0.89, 13.88, 31.53, 44.59 mean AP: 0.22723753543462002                                 
barrier Nusc dist AP@0.5, 1.0, 2.0, 4.0                                                
25.26, 46.60, 55.61, 58.85 mean AP: 0.46579271656267496                                
motorcycle Nusc dist AP@0.5, 1.0, 2.0, 4.0                                             
23.24, 28.66, 29.73, 30.35 mean AP: 0.27995502348196843                                
bicycle Nusc dist AP@0.5, 1.0, 2.0, 4.0                                                
4.74, 5.64, 6.07, 6.38 mean AP: 0.05708962692606877                                    
pedestrian Nusc dist AP@0.5, 1.0, 2.0, 4.0                                             
62.46, 64.53, 66.29, 68.61 mean AP: 0.654747663741135                                  
traffic_cone Nusc dist AP@0.5, 1.0, 2.0, 4.0                                           
42.52, 44.76, 47.14, 51.34 mean AP: 0.46438520763266                                   

[qchenclaire]

I cannot reproduce the results for nuscenes either. It took five days to train on 8 P100 even after I did some speed-up for the code. There is still difference because I use the official nuscenes devkit and ignore nan velocities. I am going to follow the exact instructions so that duplicated sweeps are trained as 0 velocities. Not sure if this will affect results.

mAP: 0.4468
mATE: 0.3524
mASE: 0.2585
mAOE: 0.7433
mAVE: 0.2495
mAAE: 0.1845
NDS: 0.5446
Eval time: 94.9s
2020-01-14 20:12:34,809 - INFO -

2020-01-14 20:12:34,813 - INFO - Evaluation nusc: Nusc v1.0-trainval Evaluation
car Nusc dist AP@0.5, 1.0, 2.0, 4.0
68.51, 80.49, 84.38, 85.78 mean AP: 0.7978895810500997
truck Nusc dist AP@0.5, 1.0, 2.0, 4.0
22.44, 42.34, 52.93, 56.83 mean AP: 0.43636250355931866
construction_vehicle Nusc dist AP@0.5, 1.0, 2.0, 4.0
0.01, 5.57, 15.38, 22.97 mean AP: 0.10984298217174057
bus Nusc dist AP@0.5, 1.0, 2.0, 4.0
31.65, 55.69, 71.26, 74.98 mean AP: 0.5839670627336495
trailer Nusc dist AP@0.5, 1.0, 2.0, 4.0
2.17, 20.15, 37.00, 48.14 mean AP: 0.2686552973499154
barrier Nusc dist AP@0.5, 1.0, 2.0, 4.0
36.64, 53.36, 59.61, 61.84 mean AP: 0.5286044968909719
motorcycle Nusc dist AP@0.5, 1.0, 2.0, 4.0
28.70, 35.14, 36.44, 36.82 mean AP: 0.3427414743865125
bicycle Nusc dist AP@0.5, 1.0, 2.0, 4.0
10.51, 11.18, 11.24, 11.42 mean AP: 0.11085860832909389
pedestrian Nusc dist AP@0.5, 1.0, 2.0, 4.0
72.51, 74.93, 76.85, 78.64 mean AP: 0.757319685027402
traffic_cone Nusc dist AP@0.5, 1.0, 2.0, 4.0
49.13, 51.24, 53.74, 58.62 mean AP: 0.5318396148928093

2020-01-14 20:12:34,818 - INFO - Epoch(val) [20][753]
2020-01-14 20:12:34,818 - INFO - task : ['car']
2020-01-14 20:12:34,818 - INFO - task : ['truck', 'construction_vehicle']
2020-01-14 20:12:34,818 - INFO - task : ['bus', 'trailer']
2020-01-14 20:12:34,818 - INFO - task : ['barrier']
2020-01-14 20:12:34,818 - INFO - task : ['motorcycle', 'bicycle']
2020-01-14 20:12:34,818 - INFO - task : ['pedestrian', 'traffic_cone']

[peiyunh]

Thanks for sharing. I also ended up ignoring NaN velocities in the smooth L1 loss, but my mAVE (#55) is much higher than yours. and got similar results. The difference is likely related to whether using the official devkit or not.

[muzi2045]

the most important metrics are mAP, mATE, mASE, mAOE, if you don't want to predict the velocity, just set velocity loss to 0, the really metric confused me is the mAOE metric. it's much larger than the report value.
By the way, there are three person try to reproduce the result, but it seems like no one get the claimed mAP value.
@peiyunh

[MeyLavie]

@peiyunh I also used nuscenes devkit provided by the authors and got very similar results:

mAP: 0.3835
mATE: 0.3727
mASE: 0.2656
mAOE: 0.8735
mAVE: 1.3856
mAAE: 0.2581
NDS: 0.4148
Eval time: 90.2s
Evaluation nusc: Nusc v1.0-trainval Evaluation
car Nusc dist AP@0.5, 1.0, 2.0, 4.0
59.47, 72.11, 77.42, 79.73 mean AP: 0.7218301684082638
truck Nusc dist AP@0.5, 1.0, 2.0, 4.0
18.09, 36.22, 45.09, 49.56 mean AP: 0.3723674852861171
construction_vehicle Nusc dist AP@0.5, 1.0, 2.0, 4.0
0.00, 2.45, 8.28, 13.50 mean AP: 0.060575765814401564
bus Nusc dist AP@0.5, 1.0, 2.0, 4.0
23.29, 47.85, 63.37, 67.68 mean AP: 0.5054781662286532
trailer Nusc dist AP@0.5, 1.0, 2.0, 4.0
1.61, 14.39, 30.38, 44.41 mean AP: 0.2269405262726112
barrier Nusc dist AP@0.5, 1.0, 2.0, 4.0
24.70, 47.08, 56.79, 59.97 mean AP: 0.47134709707812106
motorcycle Nusc dist AP@0.5, 1.0, 2.0, 4.0
23.87, 27.94, 28.82, 29.51 mean AP: 0.27537349210694445
bicycle Nusc dist AP@0.5, 1.0, 2.0, 4.0
7.67, 8.49, 9.13, 9.47 mean AP: 0.08691368416913861
pedestrian Nusc dist AP@0.5, 1.0, 2.0, 4.0
62.66, 64.41, 66.30, 68.26 mean AP: 0.654047717477926
traffic_cone Nusc dist AP@0.5, 1.0, 2.0, 4.0
42.31, 44.48, 46.41, 50.81 mean AP: 0.46001475153553095

[qchenclaire]

My recent result is

mAP: 0.4669
mATE: 0.3391
mASE: 0.2574
mAOE: 0.7657
mAVE: 0.3162
mAAE: 0.2012
NDS: 0.5455
Eval time: 89.3s
2020-01-18 08:06:51,504 - INFO - 

2020-01-18 08:06:51,505 - INFO - Evaluation nusc: Nusc v1.0-trainval Evaluation
car Nusc dist AP@0.5, 1.0, 2.0, 4.0
68.54, 80.63, 84.63, 86.58 mean AP: 0.8009424366048317
truck Nusc dist AP@0.5, 1.0, 2.0, 4.0
24.87, 44.61, 55.15, 58.78 mean AP: 0.45850578704319195
construction_vehicle Nusc dist AP@0.5, 1.0, 2.0, 4.0
0.17, 6.90, 16.21, 24.29 mean AP: 0.11890247733120854
bus Nusc dist AP@0.5, 1.0, 2.0, 4.0
32.94, 56.79, 73.28, 76.13 mean AP: 0.5978476967252525
trailer Nusc dist AP@0.5, 1.0, 2.0, 4.0
4.21, 20.78, 37.94, 53.70 mean AP: 0.29155996509713616
barrier Nusc dist AP@0.5, 1.0, 2.0, 4.0
39.77, 55.13, 59.95, 62.47 mean AP: 0.5433129255795658
motorcycle Nusc dist AP@0.5, 1.0, 2.0, 4.0
31.97, 39.27, 40.01, 40.37 mean AP: 0.3790460790422441
bicycle Nusc dist AP@0.5, 1.0, 2.0, 4.0
13.72, 14.36, 14.47, 14.65 mean AP: 0.1430066611617972
pedestrian Nusc dist AP@0.5, 1.0, 2.0, 4.0
72.81, 75.17, 76.99, 78.74 mean AP: 0.7592805073941438
traffic_cone Nusc dist AP@0.5, 1.0, 2.0, 4.0
53.54, 55.51, 58.53, 63.15 mean AP: 0.5768369346063308

After I fixed the code weight bug as @muzi2045 pointed out. I think mAOE is a bug. I found there is a little difference in direction classifier as in SECOND. I tried to disable direction classifier to see whether that would help. waiting for the results.

[qchenclaire]

But the way the code has too many bugs for lyft. I got almost 0 accuracy trying to reproduce. The code from this original repo isn't even runnable.

[muzi2045]

Why the mAVE you reproduced is normal? the others are get a larger value, could you share your modified version or config?

[zxduan90]

Why the mAVE you reproduced is normal? the others are get a larger value, could you share your modified version or config?

Maybe you should pay attention to the arg nsweep in NuscenesDataset(https://github.com/poodarchu/Det3D/blob/b4cb6428b807d8cfe2135b535961d710ff9d41c1/det3d/datasets/nuscenes/nuscenes.py#L35), its default value is 1. However the name should be n_sweep which is the same as the one in config. So I think you may train with only one lidar frame.