mAOE metric abnormal

[kaixinbear]

When reproduce detr3d, I find that whether use official provided model or train myself, the mAOE and mAOS metric is always too large.
Could someone give me any suggestion?

[a1600012888]

Seems wried.

What is the mAP and NDS of your trained model?

[kaixinbear]

Just similar to the official provided model. I don't know why.

Seems wried.

What is the mAP and NDS of your trained model?

[kaixinbear]

可以检查一下pkl里面yaw的定义跟你代码里的一致不一致。 mmdet3d在比较新的版本对坐标系做了一次调整，nuScenes的预处理和以前不一样了。如果你生成PKL的mmdet3d版本和你inference的版本不一致的话会出问题。

[a1600012888]

Thanks for the update! Got it.
New version of mmdet3d change the paramerization of orientations and some corrdinates. I don't know the details.

[yihongXU]

Hi,
Indeed, the code is not adapted to the newly released mmdet3d v1.0. For the compability, you need to update:
Basically, the changes of the new coordinate system are:

 # swap l, w (or dx, dy)
  item['gt_boxes'][:, 3] = boxes[:, 4]
  item['gt_boxes'][:, 4] = boxes[:, 3]
  # change yaw
  item['gt_boxes'][:, 6] = -boxes[:, 6] - np.pi / 2
  item['gt_boxes'][:, 6] = limit_period(
  item['gt_boxes'][:, 6], period=np.pi * 2)  

In the mmdet3d pkl files and the corresponding functions.
(c.f. https://github.com/open-mmlab/mmdetection3d/blob/master/docs/en/compatibility.md#v100rc0)

To adapt to the changes in detr3d (similar changes can be done for DGCNN):

1. for testing (the code should work for both v0.17 or v1.0 since for evaluation, we do not use pkl generated by mmdet3d):
   insert:

if int(mmdet3d.__version__[0]) >= 1: 
  #### Begin hack adaptation to mmdet3d v1.0 ####
  bboxes[:, [3, 4]] = bboxes[:, [4, 3]] 
  bboxes[:, 6] = -bboxes[:, 6] - np.pi / 2
  #### End hack adaptation to mmdet3d v1.0 ####

right after

detr3d/projects/mmdet3d_plugin/models/dense_heads/detr3d_head.py

Line 440 in 96b5606

bboxes[:, 2] = bboxes[:, 2] - bboxes[:, 5] * 0.5

You can reproduce the results as in the table.

2. For training, (not tested)
   i) You need to re-create the labels using mmdet3d >=v1.0 creat_data.py if your mmdet3d version is >=v1.0
   ii) modify the normalize_bbox function in projects/mmdet3d_plugin/core/bbox/util.py

as

def normalize_bbox(bboxes, pc_range):
    # all_bbox_preds: (cx, cy, w, l, cz, h, rot_sine, rot_cosine, vx, vy)
    # gt_bboxes_list: (cx, cy, cz, l, w, h, rot, vx, vy)
    cx = bboxes[..., 0:1]
    cy = bboxes[..., 1:2]
    cz = bboxes[..., 2:3]
    w = bboxes[..., 3:4].log()
    l = bboxes[..., 4:5].log()
    h = bboxes[..., 5:6].log()
    rot = bboxes[..., 6:7]
    
    if bboxes.size(-1) > 7:
        vx = bboxes[..., 7:8] 
        vy = bboxes[..., 8:9]

        if int(mmdet3d.__version__[0]) >= 1: 
            normalized_bboxes = torch.cat(
                (cx, cy, l, w, cz, h, (- rot - np.pi / 2).sin(), (- rot - np.pi / 2).cos(), vx, vy), dim=-1
                )
        else:
            normalized_bboxes = torch.cat(
                (cx, cy, w, l, cz, h, rot.sin(), rot.cos(), vx, vy), dim=-1
            )
    else:
        if int(mmdet3d.__version__[0]) >= 1: 
            normalized_bboxes = torch.cat(
                (cx, cy, l, w, cz, h, (- rot - np.pi / 2).sin(), (- rot - np.pi / 2).cos()), dim=-1
                )
        else:
            normalized_bboxes = torch.cat(
                (cx, cy, w, l, cz, h, rot.sin(), rot.cos()), dim=-1
            )
    return normalized_bboxes

Have fun!

[tyxsspa]

Hi @yihongXU , for training if I use mmdet3d>=1.0 to create PKL and training, why should I must change normalize_bbox function?

Hi, Indeed, the code is not adapted to the newly released mmdet3d v1.0. For the compability, you need to update: Basically, the changes of the new coordinate system are:

 # swap l, w (or dx, dy)
  item['gt_boxes'][:, 3] = boxes[:, 4]
  item['gt_boxes'][:, 4] = boxes[:, 3]
  # change yaw
  item['gt_boxes'][:, 6] = -boxes[:, 6] - np.pi / 2
  item['gt_boxes'][:, 6] = limit_period(
  item['gt_boxes'][:, 6], period=np.pi * 2)  

In the mmdet3d pkl files and the corresponding functions.

(c.f. https://github.com/open-mmlab/mmdetection3d/blob/master/docs/en/compatibility.md#v100rc0)
To adapt to the changes in detr3d (similar changes can be done for DGCNN):

1. for testing (the code should work for both v0.17 or v1.0 since for evaluation, we do not use pkl generated by mmdet3d):
   insert:

if int(mmdet3d.__version__[0]) >= 1: 
  #### Begin hack adaptation to mmdet3d v1.0 ####
  bboxes[:, [3, 4]] = bboxes[:, [4, 3]] 
  bboxes[:, 6] = -bboxes[:, 6] - np.pi / 2
  #### End hack adaptation to mmdet3d v1.0 ####

right after

detr3d/projects/mmdet3d_plugin/models/dense_heads/detr3d_head.py

Line 440 in 96b5606

bboxes[:, 2] = bboxes[:, 2] - bboxes[:, 5] * 0.5

You can reproduce the results as in the table.

2. For training, (not tested)
   i) You need to re-create the labels using mmdet3d >=v1.0 creat_data.py if your mmdet3d version is >=v1.0
   ii) modify the normalize_bbox function in projects/mmdet3d_plugin/core/bbox/util.py

as

def normalize_bbox(bboxes, pc_range):
    # all_bbox_preds: (cx, cy, w, l, cz, h, rot_sine, rot_cosine, vx, vy)
    # gt_bboxes_list: (cx, cy, cz, l, w, h, rot, vx, vy)
    cx = bboxes[..., 0:1]
    cy = bboxes[..., 1:2]
    cz = bboxes[..., 2:3]
    w = bboxes[..., 3:4].log()
    l = bboxes[..., 4:5].log()
    h = bboxes[..., 5:6].log()
    rot = bboxes[..., 6:7]
    
    if bboxes.size(-1) > 7:
        vx = bboxes[..., 7:8] 
        vy = bboxes[..., 8:9]

        if int(mmdet3d.__version__[0]) >= 1: 
            normalized_bboxes = torch.cat(
                (cx, cy, l, w, cz, h, (- rot - np.pi / 2).sin(), (- rot - np.pi / 2).cos(), vx, vy), dim=-1
                )
        else:
            normalized_bboxes = torch.cat(
                (cx, cy, w, l, cz, h, rot.sin(), rot.cos(), vx, vy), dim=-1
            )
    else:
        if int(mmdet3d.__version__[0]) >= 1: 
            normalized_bboxes = torch.cat(
                (cx, cy, l, w, cz, h, (- rot - np.pi / 2).sin(), (- rot - np.pi / 2).cos()), dim=-1
                )
        else:
            normalized_bboxes = torch.cat(
                (cx, cy, w, l, cz, h, rot.sin(), rot.cos()), dim=-1
            )
    return normalized_bboxes

Have fun!

[yihongXU]

Hi @yihongXU , for training if I use mmdet3d>=1.0 to create PKL and training, why should I must change normalize_bbox function?

Hi, Indeed, the code is not adapted to the newly released mmdet3d v1.0. For the compability, you need to update: Basically, the changes of the new coordinate system are:

 # swap l, w (or dx, dy)
  item['gt_boxes'][:, 3] = boxes[:, 4]
  item['gt_boxes'][:, 4] = boxes[:, 3]
  # change yaw
  item['gt_boxes'][:, 6] = -boxes[:, 6] - np.pi / 2
  item['gt_boxes'][:, 6] = limit_period(
  item['gt_boxes'][:, 6], period=np.pi * 2)  

In the mmdet3d pkl files and the corresponding functions.

(c.f. https://github.com/open-mmlab/mmdetection3d/blob/master/docs/en/compatibility.md#v100rc0)
To adapt to the changes in detr3d (similar changes can be done for DGCNN):

1. for testing (the code should work for both v0.17 or v1.0 since for evaluation, we do not use pkl generated by mmdet3d):
   insert:

if int(mmdet3d.__version__[0]) >= 1: 
  #### Begin hack adaptation to mmdet3d v1.0 ####
  bboxes[:, [3, 4]] = bboxes[:, [4, 3]] 
  bboxes[:, 6] = -bboxes[:, 6] - np.pi / 2
  #### End hack adaptation to mmdet3d v1.0 ####

right after

detr3d/projects/mmdet3d_plugin/models/dense_heads/detr3d_head.py

Line 440 in 96b5606

bboxes[:, 2] = bboxes[:, 2] - bboxes[:, 5] * 0.5

You can reproduce the results as in the table.

2. For training, (not tested)
   i) You need to re-create the labels using mmdet3d >=v1.0 creat_data.py if your mmdet3d version is >=v1.0
   ii) modify the normalize_bbox function in projects/mmdet3d_plugin/core/bbox/util.py

as

def normalize_bbox(bboxes, pc_range):
    # all_bbox_preds: (cx, cy, w, l, cz, h, rot_sine, rot_cosine, vx, vy)
    # gt_bboxes_list: (cx, cy, cz, l, w, h, rot, vx, vy)
    cx = bboxes[..., 0:1]
    cy = bboxes[..., 1:2]
    cz = bboxes[..., 2:3]
    w = bboxes[..., 3:4].log()
    l = bboxes[..., 4:5].log()
    h = bboxes[..., 5:6].log()
    rot = bboxes[..., 6:7]
    
    if bboxes.size(-1) > 7:
        vx = bboxes[..., 7:8] 
        vy = bboxes[..., 8:9]

        if int(mmdet3d.__version__[0]) >= 1: 
            normalized_bboxes = torch.cat(
                (cx, cy, l, w, cz, h, (- rot - np.pi / 2).sin(), (- rot - np.pi / 2).cos(), vx, vy), dim=-1
                )
        else:
            normalized_bboxes = torch.cat(
                (cx, cy, w, l, cz, h, rot.sin(), rot.cos(), vx, vy), dim=-1
            )
    else:
        if int(mmdet3d.__version__[0]) >= 1: 
            normalized_bboxes = torch.cat(
                (cx, cy, l, w, cz, h, (- rot - np.pi / 2).sin(), (- rot - np.pi / 2).cos()), dim=-1
                )
        else:
            normalized_bboxes = torch.cat(
                (cx, cy, w, l, cz, h, rot.sin(), rot.cos()), dim=-1
            )
    return normalized_bboxes

Have fun!

Hi,
It depends. Let say we want that the predictions are in the order as before, i.e.
l_bbox_preds: (cx, cy, w, l, cz, h, rot_sine, rot_cosine, vx, vy)
Depending the pkl version, the ordering of gt size is either (w, l, h) (v0.17) or (l, w, h) (v1.0). I adjust in the "normalize_bbox" (the function only used for gt bboxes), to keep the predictions ordering same as before. (The rotation is also changed...)

You can surely not to change during training, then the ordering of your predictions will also change, then you do not need to swap I did during inference.

Just be careful, the mmdet3D v1.0 expects .pkl in v1.0 because the data augmentation related to coordinates all suppose that you provide gt in (l, w, h) ordering.