Could you share the training log of stage one?

[VijayLuo]

I am reimplementing the stage one of UniAD, but the result is very different from the metrics on the repo (AMOTA: 0.3491 vs 0.390).

Could you share your training log of stage one? Thanks!

2023-04-29 17:27:50,507 - mmdet - INFO - Epoch(val) [6][753] pts_bbox_NuScenes/car_AP_dist_0.5: 0.2318, pts_bbox_NuScenes/car_AP_dist_1.0: 0.5167, pts_bbox_NuScenes/car_AP_dist_2.0: 0.7178, pts_bbox_NuScenes/car_AP_dist_4.0: 0.8132, pts_bbox_NuScenes/car_trans_err: 0.4704, pts_bbox_NuScenes/car_scale_err: 0.1501, pts_bbox_NuScenes/car_orient_err: 0.0767, pts_bbox_NuScenes/car_vel_err: 0.3336, pts_bbox_NuScenes/car_attr_err: 0.2142, pts_bbox_NuScenes/mATE: 0.6905, pts_bbox_NuScenes/mASE: 0.2765, pts_bbox_NuScenes/mAOE: 0.3952, pts_bbox_NuScenes/mAVE: 0.3878, pts_bbox_NuScenes/mAAE: 0.2061, pts_bbox_NuScenes/truck_AP_dist_0.5: 0.0345, pts_bbox_NuScenes/truck_AP_dist_1.0: 0.1856, pts_bbox_NuScenes/truck_AP_dist_2.0: 0.4547, pts_bbox_NuScenes/truck_AP_dist_4.0: 0.5935, pts_bbox_NuScenes/truck_trans_err: 0.7482, pts_bbox_NuScenes/truck_scale_err: 0.2197, pts_bbox_NuScenes/truck_orient_err: 0.0908, pts_bbox_NuScenes/truck_vel_err: 0.3117, pts_bbox_NuScenes/truck_attr_err: 0.1936, pts_bbox_NuScenes/construction_vehicle_AP_dist_0.5: 0.0000, pts_bbox_NuScenes/construction_vehicle_AP_dist_1.0: 0.0197, pts_bbox_NuScenes/construction_vehicle_AP_dist_2.0: 0.1024, pts_bbox_NuScenes/construction_vehicle_AP_dist_4.0: 0.2860, pts_bbox_NuScenes/construction_vehicle_trans_err: 0.9201, pts_bbox_NuScenes/construction_vehicle_scale_err: 0.4812, pts_bbox_NuScenes/construction_vehicle_orient_err: 1.1397, pts_bbox_NuScenes/construction_vehicle_vel_err: 0.1425, pts_bbox_NuScenes/construction_vehicle_attr_err: 0.3970, pts_bbox_NuScenes/bus_AP_dist_0.5: 0.0418, pts_bbox_NuScenes/bus_AP_dist_1.0: 0.2882, pts_bbox_NuScenes/bus_AP_dist_2.0: 0.5570, pts_bbox_NuScenes/bus_AP_dist_4.0: 0.7124, pts_bbox_NuScenes/bus_trans_err: 0.7131, pts_bbox_NuScenes/bus_scale_err: 0.2101, pts_bbox_NuScenes/bus_orient_err: 0.0862, pts_bbox_NuScenes/bus_vel_err: 0.7570, pts_bbox_NuScenes/bus_attr_err: 0.2888, pts_bbox_NuScenes/trailer_AP_dist_0.5: 0.0000, pts_bbox_NuScenes/trailer_AP_dist_1.0: 0.0151, pts_bbox_NuScenes/trailer_AP_dist_2.0: 0.1401, pts_bbox_NuScenes/trailer_AP_dist_4.0: 0.4041, pts_bbox_NuScenes/trailer_trans_err: 1.0503, pts_bbox_NuScenes/trailer_scale_err: 0.2635, pts_bbox_NuScenes/trailer_orient_err: 0.4992, pts_bbox_NuScenes/trailer_vel_err: 0.3285, pts_bbox_NuScenes/trailer_attr_err: 0.0919, pts_bbox_NuScenes/barrier_AP_dist_0.5: 0.1771, pts_bbox_NuScenes/barrier_AP_dist_1.0: 0.4155, pts_bbox_NuScenes/barrier_AP_dist_2.0: 0.6018, pts_bbox_NuScenes/barrier_AP_dist_4.0: 0.7037, pts_bbox_NuScenes/barrier_trans_err: 0.5555, pts_bbox_NuScenes/barrier_scale_err: 0.2882, pts_bbox_NuScenes/barrier_orient_err: 0.1781, pts_bbox_NuScenes/barrier_vel_err: nan, pts_bbox_NuScenes/barrier_attr_err: nan, pts_bbox_NuScenes/motorcycle_AP_dist_0.5: 0.0846, pts_bbox_NuScenes/motorcycle_AP_dist_1.0: 0.2814, pts_bbox_NuScenes/motorcycle_AP_dist_2.0: 0.5137, pts_bbox_NuScenes/motorcycle_AP_dist_4.0: 0.6110, pts_bbox_NuScenes/motorcycle_trans_err: 0.6728, pts_bbox_NuScenes/motorcycle_scale_err: 0.2607, pts_bbox_NuScenes/motorcycle_orient_err: 0.4699, pts_bbox_NuScenes/motorcycle_vel_err: 0.5474, pts_bbox_NuScenes/motorcycle_attr_err: 0.2717, pts_bbox_NuScenes/bicycle_AP_dist_0.5: 0.1006, pts_bbox_NuScenes/bicycle_AP_dist_1.0: 0.3026, pts_bbox_NuScenes/bicycle_AP_dist_2.0: 0.4610, pts_bbox_NuScenes/bicycle_AP_dist_4.0: 0.5316, pts_bbox_NuScenes/bicycle_trans_err: 0.5727, pts_bbox_NuScenes/bicycle_scale_err: 0.2676, pts_bbox_NuScenes/bicycle_orient_err: 0.5571, pts_bbox_NuScenes/bicycle_vel_err: 0.2733, pts_bbox_NuScenes/bicycle_attr_err: 0.0177, pts_bbox_NuScenes/pedestrian_AP_dist_0.5: 0.0804, pts_bbox_NuScenes/pedestrian_AP_dist_1.0: 0.3098, pts_bbox_NuScenes/pedestrian_AP_dist_2.0: 0.5757, pts_bbox_NuScenes/pedestrian_AP_dist_4.0: 0.7174, pts_bbox_NuScenes/pedestrian_trans_err: 0.7118, pts_bbox_NuScenes/pedestrian_scale_err: 0.2931, pts_bbox_NuScenes/pedestrian_orient_err: 0.4587, pts_bbox_NuScenes/pedestrian_vel_err: 0.4087, pts_bbox_NuScenes/pedestrian_attr_err: 0.1734, pts_bbox_NuScenes/traffic_cone_AP_dist_0.5: 0.2482, pts_bbox_NuScenes/traffic_cone_AP_dist_1.0: 0.4972, pts_bbox_NuScenes/traffic_cone_AP_dist_2.0: 0.6581, pts_bbox_NuScenes/traffic_cone_AP_dist_4.0: 0.7315, pts_bbox_NuScenes/traffic_cone_trans_err: 0.4896, pts_bbox_NuScenes/traffic_cone_scale_err: 0.3307, pts_bbox_NuScenes/traffic_cone_orient_err: nan, pts_bbox_NuScenes/traffic_cone_vel_err: nan, pts_bbox_NuScenes/traffic_cone_attr_err: nan, pts_bbox_NuScenes/NDS: 0.4884, pts_bbox_NuScenes/mAP: 0.3679, pts_bbox_NuScenes/amota: 0.3491, pts_bbox_NuScenes/amotp: 1.3378, pts_bbox_NuScenes/recall: 0.4500, pts_bbox_NuScenes/motar: 0.6470, pts_bbox_NuScenes/gt: 14556.7143, pts_bbox_NuScenes/mota: 0.3015, pts_bbox_NuScenes/motp: 0.7156, pts_bbox_NuScenes/mt: 2043.0000, pts_bbox_NuScenes/ml: 2778.0000, pts_bbox_NuScenes/faf: 50.1770, pts_bbox_NuScenes/tp: 54523.0000, pts_bbox_NuScenes/fp: 14212.0000, pts_bbox_NuScenes/fn: 46560.0000, pts_bbox_NuScenes/ids: 814.0000, pts_bbox_NuScenes/frag: 774.0000, pts_bbox_NuScenes/tid: 1.6311, pts_bbox_NuScenes/lgd: 2.6311, drivable_iou: 0.6575, lanes_iou: 0.2736, divider_iou: 0.2135, crossing_iou: 0.0882, contour_iou: 0.2236, drivable_iou_mean: 0.6542, lanes_iou_mean: 0.2768, divider_iou_mean: 0.2124, crossing_iou_mean: 0.0472, contour_iou_mean: 0.2361

[YTEP-ZHI]

Hi @VijayLuo, thanks for your feedback.

• Did you load the pretrained bevformer weights correctly when training stage1?

  UniAD/projects/configs/stage1_track_map/base_track_map.py

  Line 567 in 059efe4

  load_from = "ckpts/bevformer_r101_dcn_24ep.pth"

• Could you please provide the full training log in this thread so I can track the problem down?

[VijayLuo]

Hi @VijayLuo, thanks for your feedback.

• Did you load the pretrained bevformer weights correctly when training stage1?

  UniAD/projects/configs/stage1_track_map/base_track_map.py

  Line 567 in 059efe4

  load_from = "ckpts/bevformer_r101_dcn_24ep.pth"

• Could you please provide the full training log in this thread so I can track the problem down?

Thanks for your answer.

1. I did load the pretrained bevformer weights and the config is not modified.
2. Here is the whole training log

20230429_052541.log

[YTEP-ZHI]

Hi, @VijayLuo. Thanks for reporting this issue on unable to reproduce the stage1 results. We have carefully investigated this problem and found that the performance disparity is led by two causes:
1) mistakenly adding the loss_past_traj
2) freezing the img_neck and BN in stage1 training
By removing loss_past_traj and unfreezing img_neck and BN in training, the reported results could be reproduced (AMOTA: 0.393).
Here is the full training log of the fixed stage1 model: https://github.com/OpenDriveLab/UniAD/releases/download/v1.0/uniad_reproduce_stage1_gpu16_train.log

[sunbin1357]

I find that loss_past_traj is removed by setting loss_past_traj_weight to 0.0 in config，track_instances.pred_past_trajs will still be obtained during training. It may be better to add a true/false in the code to determine whether to obtain track_instances.pred_past_trajs

[YTEP-ZHI]

@sunbin1357 Thanks for your suggestion. You're right. Setting loss_past_traj_weight to 0 is an easy and correct implementation, but it could be better structured to completely remove pred_past_trajs in training. We will consider implementing it.

[YTEP-ZHI]

I'm closing this issue as it's resolved, feel free to reopen it if needed @sunbin1357 @VijayLuo.