Reproduce 3DNet+CL

[kaixinbear]

Hi @dingmyu , I have the same question here.

I want to reproduce your result in corner loss, but what you said above makes me confused.Did the corner loss useful here?How can I reproduce "3DNet+CL" ?

Originally posted by @kaixinbear in #22 (comment)

[dingmyu]

Hi,

Just remove all the depth-guided modules, set corner_loss=True and corner_in_3D=False, because only 2D corners are helpful and the 3D corner hurts the overall performance.

Regards,
Mingyu

[kaixinbear]

Hi, thanks for reply ~
I wanna to confirm twothings here:
1、I just need to set conf.use_corner = True right ?

 conf.use_corner = True
 conf.corner_in_3d = False
 conf.use_hill_loss = False

This item allows network to predict 8 corners projected cx, cy (image plane) and cz3d(camera plane).But this branch is not used in the inference stage?

2、I have some confusion about data augmentation code when doing randommirror. When the image is flipped, the vertices need to be modified like cx2d

But it seems you didn't do it in the code, which makes me confused.

Sincerely

[dingmyu]

Hi,

1. yes, it is an auxiliary loss and not used in the inference.

2. The vertex should be flipped when the image is flipped, check here https://github.com/dingmyu/D4LCN/blob/master/lib/augmentations.py#L260.

Mingyu

[kaixinbear]

2. https://github.com/dingmyu/D4LCN/blob/master/lib/augmentations.py#L260.

But it seems it only filps the center of x,have you filpped obj.vertices?

[dingmyu]

Hi, thanks for pointing out this. The vertices should be flipped as well. But it may not affect the training accuracy much (like only use half of the data for auxiliary loss).

I will do some experiments to verify this bug and update the code when I have time.

Thanks
Mingyu

[kaixinbear]

Thanks, waiting for it ! I will do some experiements either.

[kaixinbear]

Hi, dingmyu, I remove all depth ralative code and set config according to your guidance.But the performance is still low.Is my config file setting right?All could you please provide your config?

`conf: {
    model:                  resnet_dilate                         
    solver_type:            sgd                                     
    lr:                     0.004                                   
    stage2_lr:              0.001                                   
    momentum:               0.9                                     
    weight_decay:           0.0005                                  
    max_iter:               50000                                   
    snapshot_iter:          10000                                   
    display:                250                                     
    do_test:                True                                    
    use_corner:             True                                    
    corner_in_3d:           False                                   
    use_hill_loss:          False                                   
    lr_policy:              poly                                    
    lr_steps:               None                                    
    lr_target:              4e-08                                   
    rng_seed:               2                                       
    cuda_seed:              2                                       
    image_means:            [0.485, 0.456, 0.406]                   
    image_stds:             [0.229, 0.224, 0.225]                   
    feat_stride:            16                                      
    has_3d:                 True                                    
    test_scale:             512                                     
    crop_size:              [512, 1760]                             
    mirror_prob:            0.5                                     
    distort_prob:           -1                                      
    dataset_test:           kitti_split1                            
    datasets_train:         [{'anno_fmt': 'kitti_det',
                             'im_ext': '.png',
                             'name': 'kitti_split1',
                             'scale': 1}]
    use_3d_for_2d:          True                                    
    percent_anc_h:          [0.0625, 0.75]                          
    min_gt_h:               32.0                                    
    max_gt_h:               384.0                                   
    min_gt_vis:             0.65                                    
    ilbls:                  ['Van', 'ignore']                       
    lbls:                   ['Car', 'Pedestrian', 'Cyclist']        
    batch_size:             2                                       
    fg_image_ratio:         1.0                                     
    box_samples:            0.2                                     
    fg_fraction:            0.2                                     
    bg_thresh_lo:           0                                       
    bg_thresh_hi:           0.5                                     
    fg_thresh:              0.5                                     
    ign_thresh:             0.5                                     
    best_thresh:            0.35                                    
    nms_topN_pre:           3000                                    
    nms_topN_post:          40                                      
    nms_thres:              0.4                                     
    clip_boxes:             False                                   
    test_protocol:          kitti                                   
    test_db:                kitti                                   
    test_min_h:             0                                       
    min_det_scales:         [0, 0]                                  
    cluster_anchors:        0                                       
    even_anchors:           0                                       
    expand_anchors:         0                                       
    anchors:                [[-0.5, -8.5, 15.5, 23.5, 51.969, 0.531,
                             1.713, 1.025, -0.799],
                            [-8.5, -8.5, 23.5, 23.5, 52.176, 1.618,
                             1.6, 3.811, -0.453],
                            [-16.5, -8.5, 31.5, 23.5, 48.334,
                             1.644, 1.529, 3.966, 0.673],
                            [-2.528, -12.555, 17.528, 27.555,
                             44.781, 0.534, 1.771, 0.971, 0.093],
                            [-12.555, -12.555, 27.555, 27.555,
                             44.704, 1.599, 1.569, 3.814, -0.187],
                            [-22.583, -12.555, 37.583, 27.555,
                             43.492, 1.621, 1.536, 3.91, 0.719],
                            [-5.069, -17.638, 20.069, 32.638,
                             34.666, 0.561, 1.752, 0.967, -0.384],
                            [-17.638, -17.638, 32.638, 32.638,
                             35.35, 1.567, 1.591, 3.81, -0.511],
                            [-30.207, -17.638, 45.207, 32.638,
                             37.128, 1.602, 1.529, 3.904, 0.452],
                            [-8.255, -24.01, 23.255, 39.01, 28.771,
                             0.613, 1.76, 0.98, 0.067],
                            [-24.01, -24.01, 39.01, 39.01, 28.331,
                             1.543, 1.592, 3.66, -0.811],
                            [-39.764, -24.01, 54.764, 39.01,
                             30.541, 1.626, 1.524, 3.908, 0.312],
                            [-12.248, -31.996, 27.248, 46.996,
                             23.011, 0.606, 1.758, 0.996, 0.208],
                            [-31.996, -31.996, 46.996, 46.996,
                             22.948, 1.51, 1.599, 3.419, -1.076],
                            [-51.744, -31.996, 66.744, 46.996,
                             25.0, 1.628, 1.527, 3.917, 0.334],
                            [-17.253, -42.006, 32.253, 57.006,
                             18.479, 0.601, 1.747, 1.007, 0.347],
                            [-42.006, -42.006, 57.006, 57.006,
                             18.815, 1.487, 1.599, 3.337, -0.862],
                            [-66.759, -42.006, 81.759, 57.006,
                             20.576, 1.623, 1.532, 3.942, 0.323],
                            [-23.527, -54.553, 38.527, 69.553,
                             15.035, 0.625, 1.744, 0.917, 0.41],
                            [-54.553, -54.553, 69.553, 69.553,
                             15.346, 1.29, 1.659, 3.083, -0.275],
                            [-85.58, -54.553, 100.58, 69.553,
                             16.326, 1.613, 1.527, 3.934, 0.268],
                            [-31.39, -70.281, 46.39, 85.281,
                             12.265, 0.631, 1.747, 0.954, 0.317],
                            [-70.281, -70.281, 85.281, 85.281,
                             11.878, 1.044, 1.67, 2.415, -0.211],
                            [-109.171, -70.281, 124.171, 85.281,
                             13.58, 1.621, 1.539, 3.961, 0.189],
                            [-41.247, -89.994, 56.247, 104.994,
                             9.932, 0.61, 1.771, 0.934, 0.486],
                            [-89.994, -89.994, 104.994, 104.994,
                             8.949, 0.811, 1.766, 1.662, 0.08],
                            [-138.741, -89.994, 153.741, 104.994,
                             11.043, 1.61, 1.533, 3.899, 0.04],
                            [-53.602, -114.704, 68.602, 129.704,
                             8.389, 0.604, 1.793, 0.95, 0.806],
                            [-114.704, -114.704, 129.704, 129.704,
                             8.071, 1.01, 1.751, 2.19, -0.076],
                            [-175.806, -114.704, 190.806, 129.704,
                             9.184, 1.606, 1.526, 3.869, -0.066],
                            [-69.089, -145.677, 84.089, 160.677,
                             6.923, 0.627, 1.791, 0.96, 0.784],
                            [-145.677, -145.677, 160.677, 160.677,
                             6.784, 1.384, 1.615, 2.862, -1.035],
                            [-222.266, -145.677, 237.266, 160.677,
                             7.863, 1.617, 1.55, 3.948, -0.071],
                            [-88.5, -184.5, 103.5, 199.5, 5.189,
                             0.66, 1.755, 0.841, 0.173],
                            [-184.5, -184.5, 199.5, 199.5, 4.388,
                             0.743, 1.728, 1.381, 0.642],
                            [-280.5, -184.5, 295.5, 199.5, 5.583,
                             1.583, 1.547, 3.862, -0.072]]
    bbox_means:             [[-0.0, 0.002, 0.064, -0.093, 0.011,
                             -0.067, 0.192, 0.059, -0.021, 0.069,
                             -0.004, 0.19, -0.109, -0.109, -0.109,
                             -0.109, 0.045, 0.045, 0.19, -0.453,
                             -0.446, 0.329, 0.325, -0.446, -0.453,
                             0.345, 0.355, 0.06, 0.261, 0.261,
                             0.317, 0.317, 0.116, 0.116, 0.06,
                             -0.62, -0.496, -0.496, -0.595, -0.595,
                             -0.72, -0.72, -0.62, 0.066, 0.066,
                             1.654, 1.654, 0.066, 0.066, 1.654,
                             1.654, -0.608, 0.86]]
    bbox_stds:              [[0.14, 0.126, 0.255, 0.256, 0.164,
                             0.149, 3.626, 0.387, 0.104, 0.507,
                             1.855, 0.426, 0.481, 0.481, 0.413,
                             0.413, 0.49, 0.49, 0.426, 0.484,
                             0.478, 0.394, 0.392, 0.477, 0.483,
                             0.407, 0.417, 4.125, 3.825, 3.825,
                             3.855, 3.855, 4.088, 4.088, 4.125,
                             5.829, 6.051, 6.051, 5.421, 5.421,
                             5.222, 5.222, 5.829, 0.302, 0.302,
                             1.668, 1.668, 0.302, 0.302, 1.668,
                             1.668, 5.556, 0.895]]
    anchor_scales:          [32.0, 40.11, 50.276, 63.019, 78.991,
                            99.012, 124.106, 155.561, 194.989,
                            244.409, 306.354, 384.0]
    anchor_ratios:          [0.5, 1.0, 1.5]                         
    hard_negatives:         True                                    
    focal_loss:             1                                       
    cls_2d_lambda:          1                                       
    iou_2d_lambda:          1                                       
    bbox_2d_lambda:         0                                       
    bbox_3d_lambda:         1                                       
    bbox_3d_proj_lambda:    0.0                                     
    hill_climbing:          True                                    
}
[INFO]: 2020-07-17 13:26:13,307 iter: 250, acc (bg: 0.94, fg: 0.16, iou: 0.60), loss (bbox_3d: 2.1292, cls: 0.7291, iou: 0.5210, loss_vertices: 1.3247), misc (ry: 1.43, z: 2.71), dt: 0.76, eta: 10.5h
[INFO]: 2020-07-17 13:29:22,157 iter: 500, acc (bg: 1.00, fg: 0.48, iou: 0.66), loss (bbox_3d: 1.5033, cls: 0.3121, iou: 0.4334, loss_vertices: 1.3114), misc (ry: 1.24, z: 2.05), dt: 0.76, eta: 10.4h
[INFO]: 2020-07-18 00:00:06,440 iter: 49750, acc (bg: 1.00, fg: 0.98, iou: 0.93), loss (bbox_3d: 0.0498, cls: 0.0299, iou: 0.0695, loss_vertices: 1.2398), misc (ry: 0.11, z: 0.28), dt: 0.77, eta: 3.2m
[INFO]: 2020-07-18 00:03:09,989 iter: 50000, acc (bg: 1.00, fg: 0.98, iou: 0.93), loss (bbox_3d: 0.0505, cls: 0.0298, iou: 0.0702, loss_vertices: 0.9594), misc (ry: 0.12, z: 0.28), dt: 0.77, eta: 0.8s`
[INFO]: 2020-07-18 00:11:54,094 test_iter 50000 2d car --> easy: 0.8244, mod: 0.7791, hard: 0.6236
[INFO]: 2020-07-18 00:11:54,095 test_iter 50000 gr car --> easy: 0.1602, mod: 0.1113, hard: 0.0902
[INFO]: 2020-07-18 00:11:54,096 test_iter 50000 3d car --> easy: 0.1053, mod: 0.0671, hard: 0.0560
[INFO]: 2020-07-18 00:11:54,097 test_iter 50000 2d pedestrian --> easy: 0.4780, mod: 0.3982, hard: 0.3330
[INFO]: 2020-07-18 00:11:54,098 test_iter 50000 gr pedestrian --> easy: 0.0172, mod: 0.0124, hard: 0.0124
[INFO]: 2020-07-18 00:11:54,099 test_iter 50000 3d pedestrian --> easy: 0.0148, mod: 0.0108, hard: 0.0077
[INFO]: 2020-07-18 00:11:54,100 test_iter 50000 2d cyclist --> easy: 0.2836, mod: 0.1646, hard: 0.1651
[INFO]: 2020-07-18 00:11:54,100 test_iter 50000 gr cyclist --> easy: 0.0046, mod: 0.0025, hard: 0.0025
[INFO]: 2020-07-18 00:11:54,101 test_iter 50000 3d cyclist --> easy: 0.0046, mod: 0.0025, hard: 0.0025

Sincerely.

[dingmyu]

Hi,

My config file can be found here.

Your config looks normal and correct. What is your training batch size? Does the model converge in 50000 iterations?

Regards,
Mingyu

[kaixinbear]

Hi~
My training batch_size is 2, and the loss decrease in 50000 iterations except loss_vertices.The vertices loss doesn't converge. It's a bit strange for me. Do you still have the log file for this ? I think it's hard to find the bug from the config file now.

[dingmyu]

Hi,

Sorry I did not keep the log files. I think your batch size of 2 is too small so that 50000 iters are not enough for the training. I usually use 20000 iters for batch size 16, or 40000 iters for batch size 8

If you only have a single GPU, you may try a larger amount of iterations (such as 100000).

Mingyu

[kaixinbear]

Thanks,I will try it later.