Model is not training well #8

shawrby · 2022-06-14T06:23:20Z

I try to do the experiments in your paper using OpenPCDet.

For the PP-Lidar experiment, the model was trained with the default config(https://github.com/open-mmlab/OpenPCDet/blob/master/tools/cfgs/kitti_models/pointpillar.yaml), but the following results were obtained.

Compared to Car, Pedestrian and Cyclist have poor results. Especially Cyclist doesn't seem to be learning at all.
What am I missing?

andraspalffy · 2022-06-14T07:52:19Z

Please try this range:
POINT_CLOUD_RANGE: [0, -25.6, -3, 51.2, 25.6, 1]

And you should turn off the use of ground planes, we do not have those (yet).

yaoshanliang · 2022-06-14T07:57:47Z

@andraspalffy @shawrby When we use the KITTI format, it divides the dataset into Easy, Moderate and Hard. My question is that how can we get a similar AP for each class and mAP as in your paper.

andraspalffy · 2022-06-14T08:03:39Z

The evaluation code will be shared soon so you can do tests on your own. But basically, we did not use difficulties, what you see in the paper is the "Hard" category, using every object.

I think your training issues originate from the config differences, see above.

shawrby · 2022-06-14T08:37:08Z

Please try this range: POINT_CLOUD_RANGE: [0, -25.6, -3, 51.2, 25.6, 1]

And you should turn off the use of ground planes, we do not have those (yet).

@andraspalffy Thank you. Referring to your advice, I will retrain the model.

yaoshanliang · 2022-06-14T09:52:46Z

@andraspalffy How many epoches do you use in your training?

shawrby · 2022-06-14T10:08:53Z

@andraspalffy I still have the problem of model not learning well. Are there any other considerations?
@yaoshanliang Do you not have this problem?

yaoshanliang · 2022-06-14T11:14:38Z

@shawrby How many epochs do you use in your training?

shawrby · 2022-06-14T11:16:32Z

@shawrby How many epochs do you use in your training?

@yaoshanliang I used 80 epochs of the default config file.

andraspalffy · 2022-06-14T11:28:09Z

Please share both your pointpillars config and dataset config and we will check. How do you defined your splits?

shawrby · 2022-06-14T12:44:11Z

Please share both your pointpillars config and dataset config and we will check. How do you defined your splits?

@andraspalffy
These are the model config & dataset config.

Model config: pointpillar.yaml

CLASS_NAMES: ['Car', 'Pedestrian', 'Cyclist']

DATA_CONFIG: 
    _BASE_CONFIG_: jslee_cfgs/kitti_dataset.yaml
    # POINT_CLOUD_RANGE: [0, -39.68, -3, 69.12, 39.68, 1]
    POINT_CLOUD_RANGE: [0, -25.6, -3, 51.2, 25.6, 1]

    DATA_PROCESSOR:
        - NAME: mask_points_and_boxes_outside_range
          REMOVE_OUTSIDE_BOXES: True
        - NAME: shuffle_points
          SHUFFLE_ENABLED: {
            'train': True,
            'test': False
          }

        - NAME: transform_points_to_voxels
          VOXEL_SIZE: [0.16, 0.16, 4]
          MAX_POINTS_PER_VOXEL: 32
          MAX_NUMBER_OF_VOXELS: {
            'train': 16000,
            'test': 40000
          }
    DATA_AUGMENTOR:
        DISABLE_AUG_LIST: ['placeholder']
        AUG_CONFIG_LIST:
            - NAME: gt_sampling
              # USE_ROAD_PLANE: True
              USE_ROAD_PLANE: False
              DB_INFO_PATH:
                  - kitti_dbinfos_train.pkl
              PREPARE: {
                 filter_by_min_points: ['Car:5', 'Pedestrian:5', 'Cyclist:5'],
                 filter_by_difficulty: [-1],
              }

              SAMPLE_GROUPS: ['Car:15','Pedestrian:15', 'Cyclist:15']
              NUM_POINT_FEATURES: 4
              DATABASE_WITH_FAKELIDAR: False
              REMOVE_EXTRA_WIDTH: [0.0, 0.0, 0.0]
              LIMIT_WHOLE_SCENE: False

            - NAME: random_world_flip
              ALONG_AXIS_LIST: ['x']

            - NAME: random_world_rotation
              WORLD_ROT_ANGLE: [-0.78539816, 0.78539816]

            - NAME: random_world_scaling
              WORLD_SCALE_RANGE: [0.95, 1.05]

MODEL:
    NAME: PointPillar

    VFE:
        NAME: PillarVFE
        WITH_DISTANCE: False
        USE_ABSLOTE_XYZ: True
        USE_NORM: True
        NUM_FILTERS: [64]

    MAP_TO_BEV:
        NAME: PointPillarScatter
        NUM_BEV_FEATURES: 64

    BACKBONE_2D:
        NAME: BaseBEVBackbone
        LAYER_NUMS: [3, 5, 5]
        LAYER_STRIDES: [2, 2, 2]
        NUM_FILTERS: [64, 128, 256]
        UPSAMPLE_STRIDES: [1, 2, 4]
        NUM_UPSAMPLE_FILTERS: [128, 128, 128]

    DENSE_HEAD:
        NAME: AnchorHeadSingle
        CLASS_AGNOSTIC: False

        USE_DIRECTION_CLASSIFIER: True
        DIR_OFFSET: 0.78539
        DIR_LIMIT_OFFSET: 0.0
        NUM_DIR_BINS: 2

        ANCHOR_GENERATOR_CONFIG: [
            {
                'class_name': 'Car',
                'anchor_sizes': [[3.9, 1.6, 1.56]],
                'anchor_rotations': [0, 1.57],
                'anchor_bottom_heights': [-1.78],
                'align_center': False,
                'feature_map_stride': 2,
                'matched_threshold': 0.6,
                'unmatched_threshold': 0.45
            },
            {
                'class_name': 'Pedestrian',
                'anchor_sizes': [[0.8, 0.6, 1.73]],
                'anchor_rotations': [0, 1.57],
                'anchor_bottom_heights': [-0.6],
                'align_center': False,
                'feature_map_stride': 2,
                'matched_threshold': 0.5,
                'unmatched_threshold': 0.35
            },
            {
                'class_name': 'Cyclist',
                'anchor_sizes': [[1.76, 0.6, 1.73]],
                'anchor_rotations': [0, 1.57],
                'anchor_bottom_heights': [-0.6],
                'align_center': False,
                'feature_map_stride': 2,
                'matched_threshold': 0.5,
                'unmatched_threshold': 0.35
            }
        ]

        TARGET_ASSIGNER_CONFIG:
            NAME: AxisAlignedTargetAssigner
            POS_FRACTION: -1.0
            SAMPLE_SIZE: 512
            NORM_BY_NUM_EXAMPLES: False
            MATCH_HEIGHT: False
            BOX_CODER: ResidualCoder

        LOSS_CONFIG:
            LOSS_WEIGHTS: {
                'cls_weight': 1.0,
                'loc_weight': 2.0,
                'dir_weight': 0.2,
                'code_weights': [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0]
            }

    POST_PROCESSING:
        RECALL_THRESH_LIST: [0.3, 0.5, 0.7]
        SCORE_THRESH: 0.1
        OUTPUT_RAW_SCORE: False

        EVAL_METRIC: kitti

        NMS_CONFIG:
            MULTI_CLASSES_NMS: False
            NMS_TYPE: nms_gpu
            NMS_THRESH: 0.01
            NMS_PRE_MAXSIZE: 4096
            NMS_POST_MAXSIZE: 500


OPTIMIZATION:
    BATCH_SIZE_PER_GPU: 4
    NUM_EPOCHS: 80

    OPTIMIZER: adam_onecycle
    LR: 0.003
    WEIGHT_DECAY: 0.01
    MOMENTUM: 0.9

    MOMS: [0.95, 0.85]
    PCT_START: 0.4
    DIV_FACTOR: 10
    DECAY_STEP_LIST: [35, 45]
    LR_DECAY: 0.1
    LR_CLIP: 0.0000001

    LR_WARMUP: False
    WARMUP_EPOCH: 1

    GRAD_NORM_CLIP: 10

Dataset config: kitti_dataset.yaml

DATASET: 'KittiDataset'
DATA_PATH: '../data/kitti'

# POINT_CLOUD_RANGE: [0, -40, -3, 70.4, 40, 1]
POINT_CLOUD_RANGE: [0, -25.6, -3, 51.2, 25.6, 1]

DATA_SPLIT: {
    'train': train,
    'test': val
}

INFO_PATH: {
    'train': [kitti_infos_train.pkl],
    'test': [kitti_infos_val.pkl],
}

GET_ITEM_LIST: ["points"]
FOV_POINTS_ONLY: True

DATA_AUGMENTOR:
    DISABLE_AUG_LIST: ['placeholder']
    AUG_CONFIG_LIST:
        - NAME: gt_sampling
          # USE_ROAD_PLANE: True
          USE_ROAD_PLANE: False
          DB_INFO_PATH:
              - kitti_dbinfos_train.pkl
          PREPARE: {
             filter_by_min_points: ['Car:5', 'Pedestrian:5', 'Cyclist:5'],
             filter_by_difficulty: [-1],
          }

          SAMPLE_GROUPS: ['Car:20','Pedestrian:15', 'Cyclist:15']
          NUM_POINT_FEATURES: 4
          DATABASE_WITH_FAKELIDAR: False
          REMOVE_EXTRA_WIDTH: [0.0, 0.0, 0.0]
          LIMIT_WHOLE_SCENE: True

        - NAME: random_world_flip
          ALONG_AXIS_LIST: ['x']

        - NAME: random_world_rotation
          WORLD_ROT_ANGLE: [-0.78539816, 0.78539816]

        - NAME: random_world_scaling
          WORLD_SCALE_RANGE: [0.95, 1.05]


POINT_FEATURE_ENCODING: {
    encoding_type: absolute_coordinates_encoding,
    used_feature_list: ['x', 'y', 'z', 'intensity'],
    src_feature_list: ['x', 'y', 'z', 'intensity'],
}


DATA_PROCESSOR:
    - NAME: mask_points_and_boxes_outside_range
      REMOVE_OUTSIDE_BOXES: True

    - NAME: shuffle_points
      SHUFFLE_ENABLED: {
        'train': True,
        'test': False
      }

    - NAME: transform_points_to_voxels
      VOXEL_SIZE: [0.05, 0.05, 0.1]
      MAX_POINTS_PER_VOXEL: 5
      MAX_NUMBER_OF_VOXELS: {
        'train': 16000,
        'test': 40000
      }

Is there something wrong?

And dataset preparation was performed according to the OpenPCDet procedure
https://github.com/open-mmlab/OpenPCDet/blob/master/docs/GETTING_STARTED.md .

andraspalffy · 2022-06-14T13:25:58Z

This seems okay, except for the 'jslee_cfgs' config folder for base config, which I of course do not know.

Are you sure that the location you refer to '../data/kitti' has the correct data, and when you generated the splits, everything went well?

We will try to share a LiDAR training tutorial later this week.

shawrby · 2022-06-14T13:39:50Z

Thank you very much for the prompt reply.!

yaoshanliang · 2022-06-22T02:12:50Z

@andraspalffy Can you provide your config for pp-radar? I used the default config in MMDetection3D: https://github.com/open-mmlab/mmdetection3d/blob/master/configs/base/datasets/kitti-3d-3class.py

The accuracy is quite low.

shawrby · 2022-06-22T04:14:57Z

@yaoshanliang
Have you trained your model on radar data?

yaoshanliang · 2022-06-22T06:42:11Z

@shawrby Yes, the dataset root path is my radar data.

andraspalffy · 2022-06-22T09:46:23Z

Hi, we have some deadlines today and tomorrow, but we are working in the meantime to give you a nice manual of how to train similar models as we did (first LiDAR, then radar).

We did not use the mmdetection3d library, but I can see some differences already in the config:

Point cloud range has to be adapted to our dataset (especially elevation)
Voxel size has to be adapted too
Do not filter samples based on the number of points! (13th line of your default config) Radar has much fewer points than LiDAR, and this step will remove most pedestrians and cyclists.
Do use all radar features, not just the first 4 values as in the default config (line 29 of your config)
Do not use augmentations that are not compatible with radar data (e.g. rotation, see line 45 of your config, and the relevant paragraph in our paper)

Till then, here is our most recent pp-radar config for OpenPCDet, @shawrby and @yaoshanliang may be interested as well.
Note, that this has params that are not in the official OpenPCDet implementation, e.g. USE_ELEVATION.

CLASS_NAMES: ['Car', 'Pedestrian', 'Cyclist']

DATA_CONFIG: 
    _BASE_CONFIG_: cfgs/dataset_configs/radar7_as_kitti_dataset.yaml
    POINT_CLOUD_RANGE: [0, -25.6, -3, 51.2, 25.6, 2]
    DATA_PROCESSOR:
        - NAME: mask_points_and_boxes_outside_range
          REMOVE_OUTSIDE_BOXES: True

        - NAME: shuffle_points
          SHUFFLE_ENABLED: {
            'train': True, 
            'test': False
          }

        - NAME: transform_points_to_voxels
          VOXEL_SIZE: [0.16, 0.16, 5]
          MAX_POINTS_PER_VOXEL: 10
          MAX_NUMBER_OF_VOXELS: {
            'train': 16000,
            'test': 40000
          }
    DATA_AUGMENTOR:
        DISABLE_AUG_LIST: ['random_world_rotation', 'gt_sampling']
        AUG_CONFIG_LIST:

            - NAME: random_world_flip
              ALONG_AXIS_LIST: ['x']

            - NAME: random_world_scaling
              WORLD_SCALE_RANGE: [0.95, 1.05]

MODEL:
    NAME: PointPillar

    VFE:
        NAME: Radar7PillarVFE
        USE_XYZ: True
        USE_RCS: True
        USE_VR: True
        USE_VR_COMP: True
        USE_TIME: True
        USE_NORM: True
        USE_ELEVATION: True
        USE_DISTANCE: False
        NUM_FILTERS: [64]

    MAP_TO_BEV:
        NAME: PointPillarScatter
        NUM_BEV_FEATURES: 64

    BACKBONE_2D:
        NAME: BaseBEVBackbone
        LAYER_NUMS: [3, 5, 5]
        LAYER_STRIDES: [2, 2, 2]
        NUM_FILTERS: [64, 128, 256]
        UPSAMPLE_STRIDES: [1, 2, 4]
        NUM_UPSAMPLE_FILTERS: [128, 128, 128]

    DENSE_HEAD:
        NAME: AnchorHeadSingle
        CLASS_AGNOSTIC: False

        USE_DIRECTION_CLASSIFIER: True
        DIR_OFFSET: 0.78539
        DIR_LIMIT_OFFSET: 0.0
        NUM_DIR_BINS: 2

        ANCHOR_GENERATOR_CONFIG: [
            {
                'class_name': 'Car',
                'anchor_sizes': [[3.9, 1.6, 1.56]],
                'anchor_rotations': [0, 1.57],
                'anchor_bottom_heights': [-1.78],
                'align_center': False,
                'feature_map_stride': 2,
                'matched_threshold': 0.6,
                'unmatched_threshold': 0.45
            },
            {
                'class_name': 'Pedestrian',
                'anchor_sizes': [[0.8, 0.6, 1.73]],
                'anchor_rotations': [0, 1.57],
                'anchor_bottom_heights': [-0.6],
                'align_center': False,
                'feature_map_stride': 2,
                'matched_threshold': 0.5,
                'unmatched_threshold': 0.35
            },
            {
                'class_name': 'Cyclist',
                'anchor_sizes': [[1.76, 0.6, 1.73]],
                'anchor_rotations': [0, 1.57],
                'anchor_bottom_heights': [-0.6],
                'align_center': False,
                'feature_map_stride': 2,
                'matched_threshold': 0.5,
                'unmatched_threshold': 0.35
            }
        ]

        TARGET_ASSIGNER_CONFIG:
            NAME: AxisAlignedTargetAssigner
            POS_FRACTION: -1.0
            SAMPLE_SIZE: 512
            NORM_BY_NUM_EXAMPLES: False
            MATCH_HEIGHT: False
            BOX_CODER: ResidualCoder

        LOSS_CONFIG:
            LOSS_WEIGHTS: {
                'cls_weight': 1.0,
                'loc_weight': 2.0,
                'dir_weight': 0.2,
                'code_weights': [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0]
            }

    POST_PROCESSING:
        RECALL_THRESH_LIST: [0.3, 0.5, 0.7]
        SCORE_THRESH: 0.1
        OUTPUT_RAW_SCORE: False

        EVAL_METRIC: kitti

        NMS_CONFIG:
            MULTI_CLASSES_NMS: False
            NMS_TYPE: nms_gpu
            NMS_THRESH: 0.01
            NMS_PRE_MAXSIZE: 4096
            NMS_POST_MAXSIZE: 500


OPTIMIZATION:
    BATCH_SIZE_PER_GPU: 16 ##originally 16 here
    NUM_EPOCHS: 80

    OPTIMIZER: adam_onecycle
    LR: 0.003
    WEIGHT_DECAY: 0.01
    MOMENTUM: 0.9

    MOMS: [0.95, 0.85]
    PCT_START: 0.4
    DIV_FACTOR: 10
    DECAY_STEP_LIST: [35, 45]
    LR_DECAY: 0.1
    LR_CLIP: 0.0000001

    LR_WARMUP: False
    WARMUP_EPOCH: 1

    GRAD_NORM_CLIP: 10

ZixianReid · 2022-06-24T00:52:36Z

@andraspalffy Firstly, thanks to provide such a meaningful dataset. Based on your provided config file, I try to reproduction pp-lidar result. I found the training result is not good. The training result and the config file are followed.

The dataset config file:

DATASET: 'KittiDataset'
DATA_PATH: '/media/reid/ext_disk/research/radar-camera/dataset/view_of_delft_PUBLIC/lidar'

# POINT_CLOUD_RANGE: [0, -40, -3, 70.4, 40, 1]
POINT_CLOUD_RANGE: [0, -25.6, -3, 51.2, 25.6, 1]

DATA_SPLIT: {
    'train': train,
    'test': val
}

INFO_PATH: {
    'train': [kitti_infos_train.pkl],
    'test': [kitti_infos_val.pkl],
}

GET_ITEM_LIST: ["points"]
FOV_POINTS_ONLY: True

DATA_AUGMENTOR:
    DISABLE_AUG_LIST: ['placeholder']
    AUG_CONFIG_LIST:
        - NAME: gt_sampling
          # USE_ROAD_PLANE: True
          USE_ROAD_PLANE: False
          DB_INFO_PATH:
              - kitti_dbinfos_train.pkl
          PREPARE: {
             filter_by_min_points: ['Car:5', 'Pedestrian:5', 'Cyclist:5'],
             filter_by_difficulty: [-1],
          }

          SAMPLE_GROUPS: ['Car:20','Pedestrian:15', 'Cyclist:15']
          NUM_POINT_FEATURES: 4
          DATABASE_WITH_FAKELIDAR: False
          REMOVE_EXTRA_WIDTH: [0.0, 0.0, 0.0]
          LIMIT_WHOLE_SCENE: True

        - NAME: random_world_flip
          ALONG_AXIS_LIST: ['x']

        - NAME: random_world_rotation
          WORLD_ROT_ANGLE: [-0.78539816, 0.78539816]

        - NAME: random_world_scaling
          WORLD_SCALE_RANGE: [0.95, 1.05]


POINT_FEATURE_ENCODING: {
    encoding_type: absolute_coordinates_encoding,
    used_feature_list: ['x', 'y', 'z', 'intensity'],
    src_feature_list: ['x', 'y', 'z', 'intensity'],
}


DATA_PROCESSOR:
    - NAME: mask_points_and_boxes_outside_range
      REMOVE_OUTSIDE_BOXES: True

    - NAME: shuffle_points
      SHUFFLE_ENABLED: {
        'train': True,
        'test': False
      }

    - NAME: transform_points_to_voxels
      VOXEL_SIZE: [0.05, 0.05, 0.1]
      MAX_POINTS_PER_VOXEL: 5
      MAX_NUMBER_OF_VOXELS: {
        'train': 16000,
        'test': 40000
      }

The training config file:

CLASS_NAMES: ['Car', 'Pedestrian', 'Cyclist']

DATA_CONFIG:
    _BASE_CONFIG_: /home/reid/PycharmProjects/OpenPCDet/tools/cfgs/dataset_configs/kitti_dataset.yaml
    POINT_CLOUD_RANGE: [0, -25.6, -3, 51.2, 25.6, 2]
    DATA_PROCESSOR:
        - NAME: mask_points_and_boxes_outside_range
          REMOVE_OUTSIDE_BOXES: True

        - NAME: shuffle_points
          SHUFFLE_ENABLED: {
            'train': True,
            'test': False
          }

        - NAME: transform_points_to_voxels
          VOXEL_SIZE: [0.16, 0.16, 5]
          MAX_POINTS_PER_VOXEL: 10
          MAX_NUMBER_OF_VOXELS: {
            'train': 16000,
            'test': 40000
          }
    DATA_AUGMENTOR:
        DISABLE_AUG_LIST: ['random_world_rotation', 'gt_sampling']
        AUG_CONFIG_LIST:

            - NAME: random_world_flip
              ALONG_AXIS_LIST: ['x']

            - NAME: random_world_scaling
              WORLD_SCALE_RANGE: [0.95, 1.05]

MODEL:
    NAME: PointPillar


    VFE:
        NAME: PillarVFE
        WITH_DISTANCE: False
        USE_ABSLOTE_XYZ: True
        USE_NORM: True
        NUM_FILTERS: [64]

    MAP_TO_BEV:
        NAME: PointPillarScatter
        NUM_BEV_FEATURES: 64

    BACKBONE_2D:
        NAME: BaseBEVBackbone
        LAYER_NUMS: [3, 5, 5]
        LAYER_STRIDES: [2, 2, 2]
        NUM_FILTERS: [64, 128, 256]
        UPSAMPLE_STRIDES: [1, 2, 4]
        NUM_UPSAMPLE_FILTERS: [128, 128, 128]

    DENSE_HEAD:
        NAME: AnchorHeadSingle
        CLASS_AGNOSTIC: False

        USE_DIRECTION_CLASSIFIER: True
        DIR_OFFSET: 0.78539
        DIR_LIMIT_OFFSET: 0.0
        NUM_DIR_BINS: 2

        ANCHOR_GENERATOR_CONFIG: [
            {
                'class_name': 'Car',
                'anchor_sizes': [[3.9, 1.6, 1.56]],
                'anchor_rotations': [0, 1.57],
                'anchor_bottom_heights': [-1.78],
                'align_center': False,
                'feature_map_stride': 2,
                'matched_threshold': 0.6,
                'unmatched_threshold': 0.45
            },
            {
                'class_name': 'Pedestrian',
                'anchor_sizes': [[0.8, 0.6, 1.73]],
                'anchor_rotations': [0, 1.57],
                'anchor_bottom_heights': [-0.6],
                'align_center': False,
                'feature_map_stride': 2,
                'matched_threshold': 0.5,
                'unmatched_threshold': 0.35
            },
            {
                'class_name': 'Cyclist',
                'anchor_sizes': [[1.76, 0.6, 1.73]],
                'anchor_rotations': [0, 1.57],
                'anchor_bottom_heights': [-0.6],
                'align_center': False,
                'feature_map_stride': 2,
                'matched_threshold': 0.5,
                'unmatched_threshold': 0.35
            }
        ]

        TARGET_ASSIGNER_CONFIG:
            NAME: AxisAlignedTargetAssigner
            POS_FRACTION: -1.0
            SAMPLE_SIZE: 512
            NORM_BY_NUM_EXAMPLES: False
            MATCH_HEIGHT: False
            BOX_CODER: ResidualCoder

        LOSS_CONFIG:
            LOSS_WEIGHTS: {
                'cls_weight': 1.0,
                'loc_weight': 2.0,
                'dir_weight': 0.2,
                'code_weights': [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0]
            }

    POST_PROCESSING:
        RECALL_THRESH_LIST: [0.3, 0.5, 0.7]
        SCORE_THRESH: 0.1
        OUTPUT_RAW_SCORE: False

        EVAL_METRIC: kitti

        NMS_CONFIG:
            MULTI_CLASSES_NMS: False
            NMS_TYPE: nms_gpu
            NMS_THRESH: 0.01
            NMS_PRE_MAXSIZE: 4096
            NMS_POST_MAXSIZE: 500


OPTIMIZATION:
    BATCH_SIZE_PER_GPU: 4 ##originally 16 here
    NUM_EPOCHS: 80

    OPTIMIZER: adam_onecycle
    LR: 0.003
    WEIGHT_DECAY: 0.01
    MOMENTUM: 0.9

    MOMS: [0.95, 0.85]
    PCT_START: 0.4
    DIV_FACTOR: 10
    DECAY_STEP_LIST: [35, 45]
    LR_DECAY: 0.1
    LR_CLIP: 0.0000001

    LR_WARMUP: False
    WARMUP_EPOCH: 1

    GRAD_NORM_CLIP: 10

Is there any problem with my config file leading bad training results?
And I also wanna reproduction the pp-radar result. So could you also provide the pp-radar dataset config file and Radar7PillarVFE function .

shawrby · 2022-06-24T01:34:44Z

@ZixianReid
I still have the same problem..

andraspalffy · 2022-06-24T14:10:00Z

Hi all,

Thank you for your interest. I would like to emphasise that this is a dataset repo first, not a paper reproduction repo.
However, given the interest, we will make a short tutorial of how to (closely, as it is on validation test) reproduce our results next week, we are already working on it.
It is not that trivial, as data loading and feature extraction parts were slightly, but modified, thus, it is not just about sharing configs.

Hyeokreal · 2022-06-27T06:28:33Z

@andraspalffy Looking forward to it !

yaoshanliang · 2022-06-30T02:58:51Z

Could you provide the Radar7PillarVFE function? Thank you.

andraspalffy · 2022-06-30T17:41:01Z

Hi, the function will be shared in the tutorial, I do not want to further convolve this thread with half-ready codes.
Send me please an email if you are interested in the current version as it is.

andraspalffy · 2022-07-01T14:58:58Z

Dear all, good news, we think we have found the cause of the poor results.

TL;DR: please comment out this line (or equivalent step):
https://github.com/traveller59/kitti-object-eval-python/blob/master/eval.py#L69
and your results will be fine.

Longer explanation:

OpenPCDet eval (and official KITTI eval) is filtering the ground truth labels based on "Truncation" values, e.g., for the "easy" category, it removes objects that are more than 16% (0.16<truncation) truncated.
As discussed in our readme, we did not annotate the truncation levels. Instead, we used this field for other properties during our experiments, e.g. whether the object is moving (1) or not (0), or as a track id (0,1,2,3,4.....). In both cases, not a percentage but an integer.
Thus, in your evaluation, any object that has not a 0 value here (not static = moving object) was removed. This was fine for cars, because many of them are static, but for pedestrians and cyclists, very few are static (see paper), and these are often very hard cases (e.g. sitting pedestrians). Thus, poor results are expected.

Solution

Option 1: please comment out this line: https://github.com/traveller59/kitti-object-eval-python/blob/master/eval.py#L69
and your results will be fine.
Option 2: take the ground truth labels, and replace the "truncation" values with 0.
Option 3: In a couple of days we will release our version of the evaluation, which will make it very easy for you to get the results in the same format as the paper presented, and of course, handles this issue.

We applied this fix, and trained a PP-LiDAR on "train", tested on "val", and got proper results.
@shawrby @ZixianReid @ZXia221 please try this fix and get back to me. Nevertheless, please check the official evaluation script later.

@yaoshanliang This will also partly fix the radar-based training. That being said, we will provide the VFE as promised. I will open another issue for it.

ZixianReid · 2022-07-02T03:18:58Z

@andraspalffy Thanks for checking, the performance of pp-lidar is close to the paper's result now. Shown below:

2022-07-02 11:10:08,003   INFO  Car AP@0.70, 0.70, 0.70:
bbox AP:61.0117, 60.5484, 57.7078
bev  AP:64.5497, 59.2407, 57.1955
3d   AP:40.7413, 39.9368, 36.5133
aos  AP:53.70, 57.71, 54.79
Car AP_R40@0.70, 0.70, 0.70:
bbox AP:61.5815, 62.3659, 55.3603
bev  AP:63.9712, 59.5409, 54.4266
3d   AP:38.0175, 37.8173, 33.0347
aos  AP:52.95, 58.86, 52.28
Car AP@0.70, 0.50, 0.50:
bbox AP:61.0117, 60.5484, 57.7078
bev  AP:74.5348, 70.8032, 69.0569
3d   AP:70.9438, 68.1665, 60.9491
aos  AP:53.70, 57.71, 54.79
Car AP_R40@0.70, 0.50, 0.50:
bbox AP:61.5815, 62.3659, 55.3603
bev  AP:75.9536, 74.1740, 69.0067
3d   AP:71.4474, 68.3278, 61.4360
aos  AP:52.95, 58.86, 52.28
Pedestrian AP@0.50, 0.50, 0.50:
bbox AP:51.4318, 46.6757, 44.5670
bev  AP:49.0241, 43.8722, 38.3359
3d   AP:40.9660, 36.3603, 34.0826
aos  AP:37.26, 32.94, 31.54
Pedestrian AP_R40@0.50, 0.50, 0.50:
bbox AP:49.8639, 45.5566, 40.9576
bev  AP:46.2469, 41.8047, 37.3081
3d   AP:38.3998, 33.9487, 29.9051
aos  AP:33.28, 28.98, 26.10
Pedestrian AP@0.50, 0.25, 0.25:
bbox AP:51.4318, 46.6757, 44.5670
bev  AP:52.7855, 47.3169, 45.2138
3d   AP:52.6956, 47.2905, 45.1062
aos  AP:37.26, 32.94, 31.54
Pedestrian AP_R40@0.50, 0.25, 0.25:
bbox AP:49.8639, 45.5566, 40.9576
bev  AP:52.1289, 47.5596, 42.9287
3d   AP:51.0497, 47.5027, 42.8388
aos  AP:33.28, 28.98, 26.10
Cyclist AP@0.50, 0.50, 0.50:
bbox AP:71.9565, 65.0072, 61.6464
bev  AP:71.3115, 64.6404, 57.3195
3d   AP:69.3077, 63.4366, 56.4177
aos  AP:66.30, 59.34, 56.27
Cyclist AP_R40@0.50, 0.50, 0.50:
bbox AP:73.0616, 66.5190, 61.3072
bev  AP:71.6435, 64.9504, 58.2921
3d   AP:70.1385, 62.5731, 56.9279
aos  AP:66.86, 60.06, 55.17
Cyclist AP@0.50, 0.25, 0.25:
bbox AP:71.9565, 65.0072, 61.6464
bev  AP:72.4780, 65.4997, 62.4490
3d   AP:72.4593, 65.4907, 62.3581
aos  AP:66.30, 59.34, 56.27
Cyclist AP_R40@0.50, 0.25, 0.25:
bbox AP:73.0616, 66.5190, 61.3072
bev  AP:73.6565, 67.2497, 61.9781
3d   AP:73.6388, 67.2174, 61.9060
aos  AP:66.86, 60.06, 55.17

Looking forward to the pp-radar reproduced.

andraspalffy · 2022-07-04T09:20:11Z

Dear all, @BalazsSzekeres just pushed the evaluation tutorial!
d2c856e

yaoshanliang · 2022-07-10T11:53:16Z

Dear @andraspalffy, has the VFE been public?

srimannarayanabaratam · 2022-07-11T12:54:25Z

@yaoshanliang Please refer to the following document.
https://github.com/tudelft-iv/view-of-delft-dataset/blob/main/PP-Radar.md

Please consider it as a starting point and not an exhaustive list of instructions. It should however cover the major modifications we made to adapt OpenPCDet to train PointPillars on our radar point cloud.

chyohoo · 2022-08-30T02:44:38Z

@andraspalffy Can you provide your config for pp-radar? I used the default config in MMDetection3D: https://github.com/open-mmlab/mmdetection3d/blob/master/configs/_base_/datasets/kitti-3d-3class.py

The accuracy is quite low.

have you reproduced the radar results using the mmdetection3d? I was not able to obtain similar results, especially in cyclist and pedestrian.

andraspalffy · 2022-08-30T09:35:51Z

Dear @chyohoo and all users, we kindly ask you to check issues and comments before you post a new one to help our work and to make it easier for others to find answers.

Many of your questions were answered before.

Let's go over them:

Can you provide your config for pp-radar?
We did share it just above your comment. See here:
Model is not training well #8 (comment)
have you reproduced the radar results using the mmdetection3d?
No, we used OpenPCDet, as clearly explained here:
https://github.com/tudelft-iv/view-of-delft-dataset/blob/main/PP-Radar.md.
We will not be able to provide technical support for implementation, especially for other tools than we used.
The accuracy is quite low.
There are probably three reasons for that:
First, you used a default config instead of a radar-compatible config, which can be different in point cloud range, number of used features, and augmentation methods. See here:
Model is not training well #8 (comment)
Second, the reason why cyclists and pedestrians have poor performance was explained here with a solution:
Model is not training well #8 (comment)
Third, you are using the built-in eval of your library, not our published evaluation toolbox. See here:
Model is not training well #8 (comment)

Hope this helps and gets you going.
Thank you for using our dataset. Best wishes, Andras

BalazsSzekeres added the paper reproduction label Jun 14, 2022

andraspalffy closed this as completed Jul 1, 2022

andraspalffy mentioned this issue Nov 1, 2022

mmdetection3d Adaptation check #27

Closed

Lime2333 mentioned this issue May 16, 2023

Result of mAOS #45

Closed

This was referenced Jul 12, 2023

Question regarding reproducing paper results #54

Closed

How to use your assessment method, get mAP and mAOS #56

Closed

andraspalffy mentioned this issue Aug 21, 2023

How to use the evaluation function #61

Closed

This issue was closed.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Model is not training well #8

Model is not training well #8

shawrby commented Jun 14, 2022 •

edited

Loading

andraspalffy commented Jun 14, 2022 •

edited

Loading

yaoshanliang commented Jun 14, 2022

andraspalffy commented Jun 14, 2022

shawrby commented Jun 14, 2022 •

edited

Loading

yaoshanliang commented Jun 14, 2022

shawrby commented Jun 14, 2022

yaoshanliang commented Jun 14, 2022

shawrby commented Jun 14, 2022

andraspalffy commented Jun 14, 2022

shawrby commented Jun 14, 2022 •

edited

Loading

andraspalffy commented Jun 14, 2022

shawrby commented Jun 14, 2022

yaoshanliang commented Jun 22, 2022

shawrby commented Jun 22, 2022

yaoshanliang commented Jun 22, 2022

andraspalffy commented Jun 22, 2022

ZixianReid commented Jun 24, 2022

shawrby commented Jun 24, 2022

andraspalffy commented Jun 24, 2022

Hyeokreal commented Jun 27, 2022

yaoshanliang commented Jun 30, 2022

andraspalffy commented Jun 30, 2022

andraspalffy commented Jul 1, 2022

ZixianReid commented Jul 2, 2022

andraspalffy commented Jul 4, 2022

yaoshanliang commented Jul 10, 2022

srimannarayanabaratam commented Jul 11, 2022

chyohoo commented Aug 30, 2022 •

edited

Loading

andraspalffy commented Aug 30, 2022

Model is not training well #8

Model is not training well #8

Comments

shawrby commented Jun 14, 2022 • edited Loading

andraspalffy commented Jun 14, 2022 • edited Loading

yaoshanliang commented Jun 14, 2022

andraspalffy commented Jun 14, 2022

shawrby commented Jun 14, 2022 • edited Loading

yaoshanliang commented Jun 14, 2022

shawrby commented Jun 14, 2022

yaoshanliang commented Jun 14, 2022

shawrby commented Jun 14, 2022

andraspalffy commented Jun 14, 2022

shawrby commented Jun 14, 2022 • edited Loading

andraspalffy commented Jun 14, 2022

shawrby commented Jun 14, 2022

yaoshanliang commented Jun 22, 2022

shawrby commented Jun 22, 2022

yaoshanliang commented Jun 22, 2022

andraspalffy commented Jun 22, 2022

ZixianReid commented Jun 24, 2022

shawrby commented Jun 24, 2022

andraspalffy commented Jun 24, 2022

Hyeokreal commented Jun 27, 2022

yaoshanliang commented Jun 30, 2022

andraspalffy commented Jun 30, 2022

andraspalffy commented Jul 1, 2022

ZixianReid commented Jul 2, 2022

andraspalffy commented Jul 4, 2022

yaoshanliang commented Jul 10, 2022

srimannarayanabaratam commented Jul 11, 2022

chyohoo commented Aug 30, 2022 • edited Loading

andraspalffy commented Aug 30, 2022

shawrby commented Jun 14, 2022 •

edited

Loading

andraspalffy commented Jun 14, 2022 •

edited

Loading

shawrby commented Jun 14, 2022 •

edited

Loading

shawrby commented Jun 14, 2022 •

edited

Loading

chyohoo commented Aug 30, 2022 •

edited

Loading