Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Issue with KITTI "test set" evaluation turning all 0 values #1596

Closed
sinatayebati opened this issue Apr 6, 2024 · 7 comments
Closed

Issue with KITTI "test set" evaluation turning all 0 values #1596

sinatayebati opened this issue Apr 6, 2024 · 7 comments
Labels
stale

Comments

@sinatayebati
Copy link

I've successfully trained and evaluated several models using "kitti_infos_train.pkl" and "kitti_infos_val.pkl" without any problems. However, when I attempted to test using "kitti_infos_test.pkl," all the returned recall values are showing as 0!

Below, I've provided my configuration and experiment details. I would greatly appreciate it if someone could assist me in troubleshooting this issue. Thank you.

Openpcdet version: pcdet+0.6.0

Testing model: PVRCNN

  • bash scripts/dist_test.sh 2 --cfg_file cfgs/kitti_models/pv_rcnn.yaml --ckpt ../output/kitti_models/pv_rcnn/default/ckpt/checkpoint_epoch_78.pth

Evaluation result:
return _VF.meshgrid(tensors, *kwargs) # type: ignore[attr-defined]
2024-04-06 17:25:30,613 INFO ==> Loading parameters from checkpoint ../output/kitti_models/pv_rcnn_09_final/default/ckpt/checkpoint_epoch_78.pth to GPU
2024-04-06 17:25:30,695 INFO ==> Checkpoint trained from version: pcdet+0.6.0+255db8f+pybd06a67
2024-04-06 17:25:30,721 INFO ==> Done (loaded 367/367)
2024-04-06 17:25:30,740 INFO *************** EPOCH 78 EVALUATION *****************
eval: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3759/3759 [07:27<00:00, 8.40it/s, recall_0.3=(0, 0) / 0]
2024-04-06 17:32:58,430 INFO *************** Performance of EPOCH 78 *****************
2024-04-06 17:32:58,430 INFO Generate label finished(sec_per_example: 0.0595 second).
2024-04-06 17:32:58,430 INFO recall_roi_0.3: 0.000000
2024-04-06 17:32:58,430 INFO recall_rcnn_0.3: 0.000000
2024-04-06 17:32:58,430 INFO recall_roi_0.5: 0.000000
2024-04-06 17:32:58,430 INFO recall_rcnn_0.5: 0.000000
2024-04-06 17:32:58,430 INFO recall_roi_0.7: 0.000000
2024-04-06 17:32:58,430 INFO recall_rcnn_0.7: 0.000000
2024-04-06 17:32:58,441 INFO Average predicted number of objects(7518 samples): 8.798
2024-04-06 17:32:58,779 INFO None
2024-04-06 17:32:58,779 INFO Result is saved to /hdd_10tb/sina/Radial_MAE/output/kitti_models/pv_rcnn/default/eval/epoch_78/test/default
2024-04-06 17:32:58,779 INFO Evaluation done.

kitti_dataset.yaml:

DATASET: 'KittiDataset'
DATA_PATH: '../data/kitti'

POINT_CLOUD_RANGE: [0, -40, -3, 70.4, 40, 1]

DATA_SPLIT: {
'train': train,
'test': test
}

INFO_PATH: {
'train': [kitti_infos_train.pkl],
'test': [kitti_infos_test.pkl],
}

GET_ITEM_LIST: ["points"]
FOV_POINTS_ONLY: True

DATA_AUGMENTOR:
DISABLE_AUG_LIST: ['placeholder']
AUG_CONFIG_LIST:
- NAME: gt_sampling
USE_ROAD_PLANE: True
DB_INFO_PATH:
- kitti_dbinfos_train.pkl
PREPARE: {
filter_by_min_points: ['Car:5', 'Pedestrian:5', 'Cyclist:5'],
filter_by_difficulty: [-1],
}

      SAMPLE_GROUPS: ['Car:20','Pedestrian:15', 'Cyclist:15']
      NUM_POINT_FEATURES: 4
      DATABASE_WITH_FAKELIDAR: False
      REMOVE_EXTRA_WIDTH: [0.0, 0.0, 0.0]
      LIMIT_WHOLE_SCENE: True

    - NAME: random_world_flip
      ALONG_AXIS_LIST: ['x']

    - NAME: random_world_rotation
      WORLD_ROT_ANGLE: [-0.78539816, 0.78539816]

    - NAME: random_world_scaling
      WORLD_SCALE_RANGE: [0.95, 1.05]

POINT_FEATURE_ENCODING: {
encoding_type: absolute_coordinates_encoding,
used_feature_list: ['x', 'y', 'z', 'intensity'],
src_feature_list: ['x', 'y', 'z', 'intensity'],
}

DATA_PROCESSOR:
- NAME: mask_points_and_boxes_outside_range
REMOVE_OUTSIDE_BOXES: True

- NAME: shuffle_points
  SHUFFLE_ENABLED: {
    'train': True,
    'test': False
  }

- NAME: transform_points_to_voxels
  VOXEL_SIZE: [0.05, 0.05, 0.1]
  MAX_POINTS_PER_VOXEL: 5
  MAX_NUMBER_OF_VOXELS: {
    'train': 16000,
    'test': 40000
  }

pv_rcnn.yaml:

CLASS_NAMES: ['Car', 'Pedestrian', 'Cyclist']

DATA_CONFIG:
BASE_CONFIG: cfgs/dataset_configs/kitti_dataset.yaml
DATA_AUGMENTOR:
DISABLE_AUG_LIST: ['placeholder']
AUG_CONFIG_LIST:
- NAME: gt_sampling
USE_ROAD_PLANE: False
DB_INFO_PATH:
- kitti_dbinfos_train.pkl
PREPARE: {
filter_by_min_points: ['Car:5', 'Pedestrian:5', 'Cyclist:5'],
filter_by_difficulty: [-1],
}

          SAMPLE_GROUPS: ['Car:15','Pedestrian:10', 'Cyclist:10']
          NUM_POINT_FEATURES: 4
          DATABASE_WITH_FAKELIDAR: False
          REMOVE_EXTRA_WIDTH: [0.0, 0.0, 0.0]
          LIMIT_WHOLE_SCENE: False

        - NAME: random_world_flip
          ALONG_AXIS_LIST: ['x']

        - NAME: random_world_rotation
          WORLD_ROT_ANGLE: [-0.78539816, 0.78539816]

        - NAME: random_world_scaling
          WORLD_SCALE_RANGE: [0.95, 1.05]

MODEL:
NAME: PVRCNN

VFE:
    NAME: MeanVFE

BACKBONE_3D:
    NAME: VoxelBackBone8x

MAP_TO_BEV:
    NAME: HeightCompression
    NUM_BEV_FEATURES: 256

BACKBONE_2D:
    NAME: BaseBEVBackbone

    LAYER_NUMS: [5, 5]
    LAYER_STRIDES: [1, 2]
    NUM_FILTERS: [128, 256]
    UPSAMPLE_STRIDES: [1, 2]
    NUM_UPSAMPLE_FILTERS: [256, 256]

DENSE_HEAD:
    NAME: AnchorHeadSingle
    CLASS_AGNOSTIC: False

    USE_DIRECTION_CLASSIFIER: True
    DIR_OFFSET: 0.78539
    DIR_LIMIT_OFFSET: 0.0
    NUM_DIR_BINS: 2

    ANCHOR_GENERATOR_CONFIG: [
        {
            'class_name': 'Car',
            'anchor_sizes': [[3.9, 1.6, 1.56]],
            'anchor_rotations': [0, 1.57],
            'anchor_bottom_heights': [-1.78],
            'align_center': False,
            'feature_map_stride': 8,
            'matched_threshold': 0.6,
            'unmatched_threshold': 0.45
        },
        {
            'class_name': 'Pedestrian',
            'anchor_sizes': [[0.8, 0.6, 1.73]],
            'anchor_rotations': [0, 1.57],
            'anchor_bottom_heights': [-0.6],
            'align_center': False,
            'feature_map_stride': 8,
            'matched_threshold': 0.5,
            'unmatched_threshold': 0.35
        },
        {
            'class_name': 'Cyclist',
            'anchor_sizes': [[1.76, 0.6, 1.73]],
            'anchor_rotations': [0, 1.57],
            'anchor_bottom_heights': [-0.6],
            'align_center': False,
            'feature_map_stride': 8,
            'matched_threshold': 0.5,
            'unmatched_threshold': 0.35
        }
    ]

    TARGET_ASSIGNER_CONFIG:
        NAME: AxisAlignedTargetAssigner
        POS_FRACTION: -1.0
        SAMPLE_SIZE: 512
        NORM_BY_NUM_EXAMPLES: False
        MATCH_HEIGHT: False
        BOX_CODER: ResidualCoder

    LOSS_CONFIG:
        LOSS_WEIGHTS: {
            'cls_weight': 1.0,
            'loc_weight': 2.0,
            'dir_weight': 0.2,
            'code_weights': [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0]
        }

PFE:
    NAME: VoxelSetAbstraction
    POINT_SOURCE: raw_points
    NUM_KEYPOINTS: 2048
    NUM_OUTPUT_FEATURES: 128
    SAMPLE_METHOD: FPS

    FEATURES_SOURCE: ['bev', 'x_conv1', 'x_conv2', 'x_conv3', 'x_conv4', 'raw_points']
    SA_LAYER:
        raw_points:
            MLPS: [[16, 16], [16, 16]]
            POOL_RADIUS: [0.4, 0.8]
            NSAMPLE: [16, 16]
        x_conv1:
            DOWNSAMPLE_FACTOR: 1
            MLPS: [[16, 16], [16, 16]]
            POOL_RADIUS: [0.4, 0.8]
            NSAMPLE: [16, 16]
        x_conv2:
            DOWNSAMPLE_FACTOR: 2
            MLPS: [[32, 32], [32, 32]]
            POOL_RADIUS: [0.8, 1.2]
            NSAMPLE: [16, 32]
        x_conv3:
            DOWNSAMPLE_FACTOR: 4
            MLPS: [[64, 64], [64, 64]]
            POOL_RADIUS: [1.2, 2.4]
            NSAMPLE: [16, 32]
        x_conv4:
            DOWNSAMPLE_FACTOR: 8
            MLPS: [[64, 64], [64, 64]]
            POOL_RADIUS: [2.4, 4.8]
            NSAMPLE: [16, 32]

POINT_HEAD:
    NAME: PointHeadSimple
    CLS_FC: [256, 256]
    CLASS_AGNOSTIC: True
    USE_POINT_FEATURES_BEFORE_FUSION: True
    TARGET_CONFIG:
        GT_EXTRA_WIDTH: [0.2, 0.2, 0.2]
    LOSS_CONFIG:
        LOSS_REG: smooth-l1
        LOSS_WEIGHTS: {
            'point_cls_weight': 1.0,
        }

ROI_HEAD:
    NAME: PVRCNNHead
    CLASS_AGNOSTIC: True

    SHARED_FC: [256, 256]
    CLS_FC: [256, 256]
    REG_FC: [256, 256]
    DP_RATIO: 0.3

    NMS_CONFIG:
        TRAIN:
            NMS_TYPE: nms_gpu
            MULTI_CLASSES_NMS: False
            NMS_PRE_MAXSIZE: 9000
            NMS_POST_MAXSIZE: 512
            NMS_THRESH: 0.8
        TEST:
            NMS_TYPE: nms_gpu
            MULTI_CLASSES_NMS: False
            NMS_PRE_MAXSIZE: 1024
            NMS_POST_MAXSIZE: 100
            NMS_THRESH: 0.7

    ROI_GRID_POOL:
        GRID_SIZE: 6
        MLPS: [[64, 64], [64, 64]]
        POOL_RADIUS: [0.8, 1.6]
        NSAMPLE: [16, 16]
        POOL_METHOD: max_pool

    TARGET_CONFIG:
        BOX_CODER: ResidualCoder
        ROI_PER_IMAGE: 128
        FG_RATIO: 0.5

        SAMPLE_ROI_BY_EACH_CLASS: True
        CLS_SCORE_TYPE: roi_iou

        CLS_FG_THRESH: 0.75
        CLS_BG_THRESH: 0.25
        CLS_BG_THRESH_LO: 0.1
        HARD_BG_RATIO: 0.8

        REG_FG_THRESH: 0.55

    LOSS_CONFIG:
        CLS_LOSS: BinaryCrossEntropy
        REG_LOSS: smooth-l1
        CORNER_LOSS_REGULARIZATION: True
        LOSS_WEIGHTS: {
            'rcnn_cls_weight': 1.0,
            'rcnn_reg_weight': 1.0,
            'rcnn_corner_weight': 1.0,
            'code_weights': [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0]
        }

POST_PROCESSING:
    RECALL_THRESH_LIST: [0.3, 0.5, 0.7]
    SCORE_THRESH: 0.1
    OUTPUT_RAW_SCORE: False

    EVAL_METRIC: kitti

    NMS_CONFIG:
        MULTI_CLASSES_NMS: False
        NMS_TYPE: nms_gpu
        NMS_THRESH: 0.1
        NMS_PRE_MAXSIZE: 4096
        NMS_POST_MAXSIZE: 500

OPTIMIZATION:
BATCH_SIZE_PER_GPU: 2
NUM_EPOCHS: 80

OPTIMIZER: adam_onecycle
LR: 0.01
WEIGHT_DECAY: 0.01
MOMENTUM: 0.9

MOMS: [0.95, 0.85]
PCT_START: 0.4
DIV_FACTOR: 10
DECAY_STEP_LIST: [35, 45]
LR_DECAY: 0.1
LR_CLIP: 0.0000001

LR_WARMUP: False
WARMUP_EPOCH: 1

GRAD_NORM_CLIP: 10

Environment:
Package Version Editable project location

absl-py 0.15.0
addict 2.4.0
aiofiles 22.1.0
aiosqlite 0.19.0
anyio 3.7.1
appdirs 1.4.4
argon2-cffi 23.1.0
argon2-cffi-bindings 21.2.0
arrow 1.2.3
astunparse 1.6.3
attrs 23.2.0
Babel 2.14.0
backcall 0.2.0
backports.functools-lru-cache 2.0.0
beautifulsoup4 4.12.3
bleach 6.0.0
Bottleneck 1.3.5
cached-property 1.5.2
cachetools 5.3.2
ccimport 0.4.2
certifi 2024.2.2
cffi 1.15.1
charset-normalizer 3.3.2
click 8.1.7
colorama 0.4.6
comm 0.1.4
cumm-cu113 0.4.11
cycler 0.11.0
debugpy 1.6.3
decorator 5.1.1
defusedxml 0.7.1
deprecation 2.1.0
descartes 1.1.0
docker-pycreds 0.4.0
easydict 1.11
entrypoints 0.4
exceptiongroup 1.2.0
faiss-cpu 1.7.4
fastjsonschema 2.19.1
filelock 3.12.2
fire 0.5.0
flatbuffers 1.12
fonttools 4.38.0
fqdn 1.5.1
fvcore 0.1.5.post20210915
gast 0.4.0
gdown 4.4.0
gitdb 4.0.11
GitPython 3.1.41
google-auth 2.27.0
google-auth-oauthlib 0.4.6
google-pasta 0.2.0
grpcio 1.34.1
h5py 3.7.0
idna 3.6
imageio 2.31.2
importlib-metadata 6.7.0
importlib-resources 5.12.0
iopath 0.1.9
ipykernel 6.16.2
ipython 7.33.0
ipython-genutils 0.2.0
ipywidgets 8.1.2
isoduration 20.11.0
jedi 0.19.1
Jinja2 3.1.3
joblib 1.3.2
json5 0.9.14
jsonpointer 2.4
jsonschema 4.17.3
jupyter 1.0.0
jupyter_client 7.4.9
jupyter-console 6.6.3
jupyter_core 4.12.0
jupyter-events 0.5.0
jupyter_packaging 0.12.3
jupyter-server 1.24.0
jupyter_server_fileid 0.9.1
jupyter_server_ydoc 0.8.0
jupyter-ydoc 0.2.5
jupyterlab 3.6.7
jupyterlab-pygments 0.2.2
jupyterlab_server 2.24.0
jupyterlab_widgets 3.0.10
keras-nightly 2.5.0.dev2021032900
Keras-Preprocessing 1.1.2
kiwisolver 1.4.5
lark 1.1.9
llvmlite 0.39.1
Markdown 3.4.4
MarkupSafe 2.1.5
matplotlib 3.5.2
matplotlib-inline 0.1.3
mistune 3.0.2
motmetrics 1.4.0
nbclassic 1.0.0
nbclient 0.7.0
nbconvert 7.6.0
nbformat 5.8.0
nest_asyncio 1.6.0
networkx 2.6.3
ninja 1.11.1.1
notebook 6.5.6
notebook_shim 0.2.3
numba 0.56.4
numexpr 2.7.3
numpy 1.19.5
nuscenes-devkit 1.0.5
oauthlib 3.2.2
open3d 0.15.2
opencv-python 4.6.0.66
opt-einsum 3.3.0
packaging 23.2
pandas 1.3.5
pandocfilters 1.5.1
parso 0.8.3
patsy 0.5.6
pccm 0.4.11
pcdet 0.6.0+255db8f /hdd_10tb/sina/Radial_MAE
pexpect 4.9.0
pickleshare 0.7.5
Pillow 9.2.0
pip 24.0
pkgutil_resolve_name 1.3.10
plotly 5.8.1
plotly-express 0.4.1
portalocker 2.6.0
prometheus-client 0.17.1
prompt-toolkit 3.0.42
protobuf 3.20.1
psutil 5.9.3
ptyprocess 0.7.0
pyasn1 0.5.1
pyasn1-modules 0.3.0
pybind11 2.11.1
pycparser 2.21
Pygments 2.17.2
pyparsing 3.1.1
pyquaternion 0.9.9
pyrsistent 0.19.3
PySocks 1.7.1
python-dateutil 2.8.2
python-json-logger 2.0.7
pytorch3d 0.6.2
pytz 2024.1
PyWavelets 1.3.0
PyYAML 6.0
pyzmq 24.0.1
qtconsole 5.4.4
QtPy 2.4.1
requests 2.31.0
requests-oauthlib 1.3.1
rfc3339-validator 0.1.4
rfc3986-validator 0.1.1
rsa 4.9
scikit-image 0.19.3
scikit-learn 1.0.2
scipy 1.5.3
Send2Trash 1.8.2
sentry-sdk 1.40.3
setproctitle 1.3.3
setuptools 62.3.3
shapely 2.0.3
SharedArray 3.1.0
six 1.15.0
smmap 5.0.1
sniffio 1.3.0
soupsieve 2.4.1
spconv-cu113 2.3.6
statsmodels 0.13.2
tabulate 0.9.0
tenacity 8.2.3
tensorboard 2.11.2
tensorboard-data-server 0.6.1
tensorboard-plugin-wit 1.8.1
tensorboardX 2.5.1
tensorflow 2.5.0
tensorflow-estimator 2.5.0
termcolor 1.1.0
terminado 0.17.1
threadpoolctl 3.1.0
tifffile 2021.11.2
timm 0.4.5
tinycss2 1.1.1
tomli 2.0.1
tomlkit 0.11.0
torch 1.10.2
torch-scatter 2.1.1
torchvision 0.11.3
tornado 6.2
tqdm 4.64.0
traitlets 5.9.0
transforms3d 0.3.1
typing_extensions 4.7.1
uri-template 1.3.0
urllib3 2.0.7
wandb 0.16.3
waymo-open-dataset-tf-2-5-0 1.4.1
wcwidth 0.2.10
webcolors 1.13
webencodings 0.5.1
websocket-client 1.6.1
Werkzeug 2.2.3
wheel 0.42.0
widgetsnbextension 4.0.10
wrapt 1.12.1
xmltodict 0.13.0
y-py 0.6.2
yacs 0.1.8
ypy-websocket 0.8.4
zipp 3.15.0
@Dai-Yu-Chen
Copy link

Hi, I have the same problem. Have you solved this problem?

@sinatayebati
Copy link
Author

Hi, I have the same problem. Have you solved this problem?

Unfortunately not!

@github-actions GitHub Actions
Copy link

This issue is stale because it has been open for 30 days with no activity.

@github-actions github-actions bot added the stale label May 12, 2024
@calcualatexzy
Copy link

It is very annoying that I saw the same problem in "issues" several times and the team keeps on closing the issues due to 30 days stale, so is there any solution?

@github-actions github-actions bot removed the stale label May 15, 2024
@josyulavt
Copy link

I have the same problem during training the loss is shown to be 0.2, I wanted to see the test results and everything is 0.

Any help is much appreciated

@github-actions GitHub Actions
Copy link

This issue is stale because it has been open for 30 days with no activity.

@github-actions github-actions bot added the stale label Jun 17, 2024
@github-actions GitHub Actions
Copy link

github-actions bot commented Jul 1, 2024

This issue was closed because it has been inactive for 14 days since being marked as stale.

@github-actions github-actions bot closed this as completed Jul 1, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
stale
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@sinatayebati @calcualatexzy @Dai-Yu-Chen @josyulavt