Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

KeyError: 'type #2

Closed
chyohoo opened this issue Jan 4, 2022 · 14 comments
Closed

KeyError: 'type #2

chyohoo opened this issue Jan 4, 2022 · 14 comments

Comments

@chyohoo
Copy link

chyohoo commented Jan 4, 2022

Hi, thanks for this fantastic work. I have met a problem seeing blow.

Traceback (most recent call last):
  File "tools/train.py", line 230, in <module>
    main()
  File "tools/train.py", line 194, in main
    datasets = [build_dataset(cfg.data.train)]
  File "/home/chen_haoye/WS_SST/SST/mmdet3d/datasets/builder.py", line 32, in build_dataset
    build_dataset(cfg['dataset'], default_args), cfg['times'])
  File "/home/chen_haoye/WS_SST/SST/mmdet3d/datasets/builder.py", line 26, in build_dataset
    elif cfg['type'] == 'ConcatDataset':
  File "/home/chen_haoye/anaconda3/envs/SST/lib/python3.7/site-packages/mmcv/utils/config.py", line 36, in __missing__
    raise KeyError(name)
KeyError: 'type'

I have printed the keys in cfg, but the cfg did have the type key. What could be wrong?
@Abyssaledge
Copy link
Collaborator

Thanks for using SST. I need more information to identify this error: Which config do you use? Do you modify any codes? And what commands do you run?

@chyohoo
Copy link
Author

chyohoo commented Jan 4, 2022

Hi, I was trying to use SST on nuScenes radar data. I only modified the config and nuScenes converter.
and I ran this CONFIG=sst_nuscenes bash tools/dist_train.sh configs/sst/$CONFIG.py 2 --work-dir ./work_dirs/$CONFIG/ --cfg-options evaluation.pklfile_prefix=./work_dirs/$CONFIG/results evaluation.metric=bbox
Here is the config I use
`Config:
model = dict(
type='DynamicVoxelNet',
neck=dict(
type='SECONDFPN',
norm_cfg=dict(type='naiveSyncBN2d', eps=0.001, momentum=0.01),
in_channels=[128],
upsample_strides=[1],
out_channels=[384]),
bbox_head=dict(
type='Anchor3DHead',
num_classes=10,
in_channels=384,
feat_channels=384,
use_direction_classifier=True,
anchor_generator=dict(
type='AlignedAnchor3DRangeGenerator',
ranges=[[0.4, -49.6, -1.80032795, 49.6, 49.6, -1.80032795],
[0.4, -49.6, -1.74440365, 49.6, 49.6, -1.74440365],
[0.4, -49.6, -1.68526504, 49.6, 49.6, -1.68526504],
[0.4, -49.6, -1.74440365, 49.6, 49.6, -1.74440365],
[0.4, -49.6, -1.74440365, 49.6, 49.6, -1.74440365],
[0.4, -49.6, -1.67339111, 49.6, 49.6, -1.67339111],
[0.4, -49.6, -1.67339111, 49.6, 49.6, -1.67339111],
[0.4, -49.6, -1.61785072, 49.6, 49.6, -1.61785072],
[0.4, -49.6, -1.80984986, 49.6, 49.6, -1.80984986],
[0.4, -49.6, -1.763965, 49.6, 49.6, -1.763965]],
sizes=[[1.95017717, 4.60718145, 1.72270761],
[2.4560939, 6.73778078, 2.73004906],
[2.87427237, 12.01320693, 3.81509561],
[2.94046906, 11.1885991, 3.47030982],
[2.73050468, 6.38352896, 3.13312415],
[0.60058911, 1.68452161, 1.27192197],
[0.76279481, 2.09973778, 1.44403034],
[0.66344886, 0.7256437, 1.75748069],
[0.39694519, 0.40359262, 1.06232151],
[2.49008838, 0.48578221, 0.98297065]],
rotations=[0, 1.57],
reshape_out=False),
diff_rad_by_sin=True,
dir_offset=0.7854,
dir_limit_offset=0,
bbox_coder=dict(type='DeltaXYZWLHRBBoxCoder', code_size=7),
loss_cls=dict(
type='FocalLoss',
use_sigmoid=True,
gamma=2.0,
alpha=0.25,
loss_weight=1.0),
loss_bbox=dict(type='L1Loss', loss_weight=0.5),
loss_dir=dict(
type='CrossEntropyLoss', use_sigmoid=False, loss_weight=0.2)),
train_cfg=dict(
assigner=[
dict(
type='MaxIoUAssigner',
iou_calculator=dict(type='BboxOverlapsNearest3D'),
pos_iou_thr=0.5,
neg_iou_thr=0.3,
min_pos_iou=0.3,
ignore_iof_thr=-1),
dict(
type='MaxIoUAssigner',
iou_calculator=dict(type='BboxOverlapsNearest3D'),
pos_iou_thr=0.5,
neg_iou_thr=0.3,
min_pos_iou=0.3,
ignore_iof_thr=-1)
],
allowed_border=0,
code_weight=[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
pos_weight=-1,
debug=False),
test_cfg=dict(
use_rotate_nms=True,
nms_across_levels=False,
nms_pre=4096,
nms_thr=0.25,
score_thr=0.1,
min_bbox_size=0,
max_num=500),
voxel_layer=dict(
voxel_size=(0.2, 0.2, 6),
max_num_points=-1,
point_cloud_range=[0, -50, -5, 50, 50, 3],
max_voxels=(-1, -1)),
voxel_encoder=dict(
type='DynamicVFE',
in_channels=7,
feat_channels=[64, 128],
with_distance=False,
voxel_size=(0.2, 0.2, 6),
with_cluster_center=True,
with_voxel_center=True,
point_cloud_range=[0, -50, -5, 50, 50, 3],
norm_cfg=dict(type='naiveSyncBN1d', eps=0.001, momentum=0.01)),
middle_encoder=dict(
type='SSTInputLayer',
window_shape=(10, 10),
shifts_list=[(0, 0), (5, 5)],
point_cloud_range=[0, -50, -5, 50, 50, 3],
voxel_size=(0.2, 0.2, 6),
shuffle_voxels=True,
debug=True,
drop_info=({
0: {
'max_tokens': 30,
'drop_range': (0, 30)
},
1: {
'max_tokens': 60,
'drop_range': (30, 60)
},
2: {
'max_tokens': 100,
'drop_range': (60, 100000)
}
}, {
0: {
'max_tokens': 30,
'drop_range': (0, 30)
},
1: {
'max_tokens': 60,
'drop_range': (30, 60)
},
2: {
'max_tokens': 100,
'drop_range': (60, 100000)
}
})),
backbone=dict(
type='SSTv1',
d_model=[128, 128, 128, 128, 128, 128],
nhead=[8, 8, 8, 8, 8, 8],
num_blocks=6,
dim_feedforward=[256, 256, 256, 256, 256, 256],
output_shape=[468, 468],
num_attached_conv=3,
conv_kwargs=[
dict(kernel_size=3, dilation=1, padding=1, stride=1),
dict(kernel_size=3, dilation=1, padding=1, stride=1),
dict(kernel_size=3, dilation=1, padding=1, stride=1)
],
conv_in_channel=128,
conv_out_channel=128,
debug=True,
drop_info=({
0: {
'max_tokens': 30,
'drop_range': (0, 30)
},
1: {
'max_tokens': 60,
'drop_range': (30, 60)
},
2: {
'max_tokens': 100,
'drop_range': (60, 100000)
}
}, {
0: {
'max_tokens': 30,
'drop_range': (0, 30)
},
1: {
'max_tokens': 60,
'drop_range': (30, 60)
},
2: {
'max_tokens': 100,
'drop_range': (60, 100000)
}
}),
pos_temperature=10000,
normalize_pos=False,
window_shape=(10, 10),
checkpoint_blocks=[0, 1, 2]))
point_cloud_range = [0, -50, -5, 50, 50, 3]
class_names = [
'car', 'truck', 'trailer', 'bus', 'construction_vehicle', 'bicycle',
'motorcycle', 'pedestrian', 'traffic_cone', 'barrier'
]
dataset_type = 'NuScenesDatasetRadar'
data_root = '/home/algo_zf/Data/nuscence'
input_modality = dict(
use_lidar=True,
use_camera=False,
use_radar=False,
use_map=False,
use_external=False)
file_client_args = dict(backend='disk')
train_pipeline = [
dict(
type='LoadPointsFromFile',
coord_type='LIDAR',
load_dim=18,
use_dim=18,
file_client_args=dict(backend='disk')),
dict(
type='LoadPointsFromMultiSweeps',
sweeps_num=10,
use_dim=[0, 1, 2, 3, 5, 8, 9],
pad_empty_sweeps=True,
file_client_args=dict(backend='disk')),
dict(type='LoadAnnotations3D', with_bbox_3d=True, with_label_3d=True),
dict(
type='GlobalRotScaleTrans',
rot_range=[-0.3925, 0.3925],
scale_ratio_range=[0.95, 1.05],
translation_std=[0, 0, 0]),
dict(type='RandomFlip3D', flip_ratio_bev_horizontal=0.5),
dict(type='PointsRangeFilter', point_cloud_range=[0, -50, -5, 50, 50, 3]),
dict(type='ObjectRangeFilter', point_cloud_range=[0, -50, -5, 50, 50, 3]),
dict(
type='ObjectNameFilter',
classes=[
'car', 'truck', 'trailer', 'bus', 'construction_vehicle',
'bicycle', 'motorcycle', 'pedestrian', 'traffic_cone', 'barrier'
]),
dict(type='PointShuffle'),
dict(
type='DefaultFormatBundle3D',
class_names=[
'car', 'truck', 'trailer', 'bus', 'construction_vehicle',
'bicycle', 'motorcycle', 'pedestrian', 'traffic_cone', 'barrier'
]),
dict(type='Collect3D', keys=['points', 'gt_bboxes_3d', 'gt_labels_3d'])
]
test_pipeline = [
dict(
type='LoadPointsFromFile',
coord_type='LIDAR',
load_dim=18,
use_dim=18,
file_client_args=dict(backend='disk')),
dict(
type='LoadPointsFromMultiSweeps',
sweeps_num=10,
use_dim=[0, 1, 2, 3, 5, 8, 9],
pad_empty_sweeps=True,
file_client_args=dict(backend='disk')),
dict(
type='MultiScaleFlipAug3D',
img_scale=(1333, 800),
pts_scale_ratio=1,
flip=False,
transforms=[
dict(
type='GlobalRotScaleTrans',
rot_range=[0, 0],
scale_ratio_range=[1.0, 1.0],
translation_std=[0, 0, 0]),
dict(type='RandomFlip3D'),
dict(
type='PointsRangeFilter',
point_cloud_range=[0, -50, -5, 50, 50, 3]),
dict(
type='DefaultFormatBundle3D',
class_names=[
'car', 'truck', 'trailer', 'bus', 'construction_vehicle',
'bicycle', 'motorcycle', 'pedestrian', 'traffic_cone',
'barrier'
],
with_label=False),
dict(type='Collect3D', keys=['points'])
])
]
eval_pipeline = [
dict(
type='LoadPointsFromFile',
coord_type='LIDAR',
load_dim=18,
use_dim=18,
file_client_args=dict(backend='disk')),
dict(
type='LoadPointsFromMultiSweeps',
sweeps_num=10,
use_dim=[0, 1, 2, 3, 5, 8, 9],
pad_empty_sweeps=True,
file_client_args=dict(backend='disk')),
dict(
type='DefaultFormatBundle3D',
class_names=[
'car', 'truck', 'trailer', 'bus', 'construction_vehicle',
'bicycle', 'motorcycle', 'pedestrian', 'traffic_cone', 'barrier'
],
with_label=False),
dict(type='Collect3D', keys=['points'])
]
data = dict(
samples_per_gpu=1,
workers_per_gpu=4,
train=dict(
type='RepeatDataset',
data_root='/home/algo_zf/Data/nuscence',
ann_file='/home/algo_zf/Data/nuscencenuscenes_infos_train.pkl',
pipeline=[
dict(
type='LoadPointsFromFile',
coord_type='LIDAR',
load_dim=18,
use_dim=18,
file_client_args=dict(backend='disk')),
dict(
type='LoadPointsFromMultiSweeps',
sweeps_num=10,
use_dim=[0, 1, 2, 3, 5, 8, 9],
pad_empty_sweeps=True,
file_client_args=dict(backend='disk')),
dict(
type='LoadAnnotations3D',
with_bbox_3d=True,
with_label_3d=True),
dict(
type='GlobalRotScaleTrans',
rot_range=[-0.3925, 0.3925],
scale_ratio_range=[0.95, 1.05],
translation_std=[0, 0, 0]),
dict(type='RandomFlip3D', flip_ratio_bev_horizontal=0.5),
dict(
type='PointsRangeFilter',
point_cloud_range=[0, -50, -5, 50, 50, 3]),
dict(
type='ObjectRangeFilter',
point_cloud_range=[0, -50, -5, 50, 50, 3]),
dict(
type='ObjectNameFilter',
classes=[
'car', 'truck', 'trailer', 'bus', 'construction_vehicle',
'bicycle', 'motorcycle', 'pedestrian', 'traffic_cone',
'barrier'
]),
dict(type='PointShuffle'),
dict(
type='DefaultFormatBundle3D',
class_names=[
'car', 'truck', 'trailer', 'bus', 'construction_vehicle',
'bicycle', 'motorcycle', 'pedestrian', 'traffic_cone',
'barrier'
]),
dict(
type='Collect3D',
keys=['points', 'gt_bboxes_3d', 'gt_labels_3d'])
],
classes=[
'car', 'truck', 'trailer', 'bus', 'construction_vehicle',
'bicycle', 'motorcycle', 'pedestrian', 'traffic_cone', 'barrier'
],
modality=dict(
use_lidar=True,
use_camera=False,
use_radar=False,
use_map=False,
use_external=False),
test_mode=False,
box_type_3d='LiDAR',
times=1,
dataset=dict(load_interval=10)),
val=dict(
type='NuScenesDatasetRadar',
data_root='/home/algo_zf/Data/nuscence',
ann_file='/home/algo_zf/Data/nuscencenuscenes_infos_val.pkl',
pipeline=[
dict(
type='LoadPointsFromFile',
coord_type='LIDAR',
load_dim=18,
use_dim=18,
file_client_args=dict(backend='disk')),
dict(
type='LoadPointsFromMultiSweeps',
sweeps_num=10,
use_dim=[0, 1, 2, 3, 5, 8, 9],
pad_empty_sweeps=True,
file_client_args=dict(backend='disk')),
dict(
type='MultiScaleFlipAug3D',
img_scale=(1333, 800),
pts_scale_ratio=1,
flip=False,
transforms=[
dict(
type='GlobalRotScaleTrans',
rot_range=[0, 0],
scale_ratio_range=[1.0, 1.0],
translation_std=[0, 0, 0]),
dict(type='RandomFlip3D'),
dict(
type='PointsRangeFilter',
point_cloud_range=[0, -50, -5, 50, 50, 3]),
dict(
type='DefaultFormatBundle3D',
class_names=[
'car', 'truck', 'trailer', 'bus',
'construction_vehicle', 'bicycle', 'motorcycle',
'pedestrian', 'traffic_cone', 'barrier'
],
with_label=False),
dict(type='Collect3D', keys=['points'])
])
],
classes=[
'car', 'truck', 'trailer', 'bus', 'construction_vehicle',
'bicycle', 'motorcycle', 'pedestrian', 'traffic_cone', 'barrier'
],
modality=dict(
use_lidar=True,
use_camera=False,
use_radar=False,
use_map=False,
use_external=False),
test_mode=True,
box_type_3d='LiDAR'),
test=dict(
type='NuScenesDatasetRadar',
data_root='/home/algo_zf/Data/nuscence',
ann_file='/home/algo_zf/Data/nuscencenuscenes_infos_val.pkl',
pipeline=[
dict(
type='LoadPointsFromFile',
coord_type='LIDAR',
load_dim=18,
use_dim=18,
file_client_args=dict(backend='disk')),
dict(
type='LoadPointsFromMultiSweeps',
sweeps_num=10,
use_dim=[0, 1, 2, 3, 5, 8, 9],
pad_empty_sweeps=True,
file_client_args=dict(backend='disk')),
dict(
type='MultiScaleFlipAug3D',
img_scale=(1333, 800),
pts_scale_ratio=1,
flip=False,
transforms=[
dict(
type='GlobalRotScaleTrans',
rot_range=[0, 0],
scale_ratio_range=[1.0, 1.0],
translation_std=[0, 0, 0]),
dict(type='RandomFlip3D'),
dict(
type='PointsRangeFilter',
point_cloud_range=[0, -50, -5, 50, 50, 3]),
dict(
type='DefaultFormatBundle3D',
class_names=[
'car', 'truck', 'trailer', 'bus',
'construction_vehicle', 'bicycle', 'motorcycle',
'pedestrian', 'traffic_cone', 'barrier'
],
with_label=False),
dict(type='Collect3D', keys=['points'])
])
],
classes=[
'car', 'truck', 'trailer', 'bus', 'construction_vehicle',
'bicycle', 'motorcycle', 'pedestrian', 'traffic_cone', 'barrier'
],
modality=dict(
use_lidar=True,
use_camera=False,
use_radar=False,
use_map=False,
use_external=False),
test_mode=True,
box_type_3d='LiDAR'))
evaluation = dict(
interval=12,
pipeline=[
dict(
type='LoadPointsFromFile',
coord_type='LIDAR',
load_dim=18,
use_dim=18,
file_client_args=dict(backend='disk')),
dict(
type='LoadPointsFromMultiSweeps',
sweeps_num=10,
use_dim=[0, 1, 2, 3, 5, 8, 9],
pad_empty_sweeps=True,
file_client_args=dict(backend='disk')),
dict(
type='DefaultFormatBundle3D',
class_names=[
'car', 'truck', 'trailer', 'bus', 'construction_vehicle',
'bicycle', 'motorcycle', 'pedestrian', 'traffic_cone',
'barrier'
],
with_label=False),
dict(type='Collect3D', keys=['points'])
])
lr = 1e-05
optimizer = dict(
type='AdamW',
lr=1e-05,
betas=(0.9, 0.999),
weight_decay=0.05,
paramwise_cfg=dict(custom_keys=dict(norm=dict(decay_mult=0.0))))
optimizer_config = dict(grad_clip=dict(max_norm=10, norm_type=2))
lr_config = dict(
policy='cyclic',
target_ratio=(100, 0.001),
cyclic_times=1,
step_ratio_up=0.1)
momentum_config = None
runner = dict(type='EpochBasedRunner', max_epochs=12)
checkpoint_config = dict(interval=1)
log_config = dict(
interval=50,
hooks=[dict(type='TextLoggerHook'),
dict(type='TensorboardLoggerHook')])
dist_params = dict(backend='nccl')
log_level = 'INFO'
work_dir = None
load_from = None
resume_from = None
workflow = [('train', 1)]
voxel_size = (0.2, 0.2, 6)
window_shape = (10, 10)
drop_info_training = dict({
0: dict(max_tokens=30, drop_range=(0, 30)),
1: dict(max_tokens=60, drop_range=(30, 60)),
2: dict(max_tokens=100, drop_range=(60, 100000))
})
drop_info_test = dict({
0: dict(max_tokens=30, drop_range=(0, 30)),
1: dict(max_tokens=60, drop_range=(30, 60)),
2: dict(max_tokens=100, drop_range=(60, 100000))
})
drop_info = ({
0: {
'max_tokens': 30,
'drop_range': (0, 30)
},
1: {
'max_tokens': 60,
'drop_range': (30, 60)
},
2: {
'max_tokens': 100,
'drop_range': (60, 100000)
}
}, {
0: {
'max_tokens': 30,
'drop_range': (0, 30)
},
1: {
'max_tokens': 60,
'drop_range': (30, 60)
},
2: {
'max_tokens': 100,
'drop_range': (60, 100000)
}
})
shifts_list = [(0, 0), (5, 5)]
fp16 = dict(loss_scale=32.0)

`

@chyohoo
Copy link
Author

chyohoo commented Jan 4, 2022 •
edited

the running env is

sys.platform: linux
Python: 3.7.12 | packaged by conda-forge | (default, Oct 26 2021, 06:08:21) [GCC 9.4.0]
CUDA available: True
GPU 0,1: Quadro RTX 8000
CUDA_HOME: /usr/local/cuda
NVCC: Cuda compilation tools, release 10.2, V10.2.89
GCC: gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
PyTorch: 1.10.1
PyTorch compiling details: PyTorch built with:

  • GCC 7.3
  • C++ Version: 201402
  • Intel(R) Math Kernel Library Version 2020.0.4 Product Build 20200917 for Intel(R) 64 architecture applications
  • Intel(R) MKL-DNN v2.2.3 (Git Hash 7336ca9f055cf1bfa13efb658fe15dc9b41f0740)
  • OpenMP 201511 (a.k.a. OpenMP 4.5)
  • LAPACK is enabled (usually provided by MKL)
  • NNPACK is enabled
  • CPU capability usage: AVX512
  • CUDA Runtime 10.2
  • NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_61,code=sm_61;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_37,code=compute_37
  • CuDNN 7.6.5
  • Magma 2.5.2
  • Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=10.2, CUDNN_VERSION=7.6.5, CXX_COMPILER=/opt/rh/devtoolset-7/root/usr/bin/c++, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -DEDGE_PROFILER_USE_KINETO -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=1.10.1, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON,

TorchVision: 0.11.2
OpenCV: 4.5.5
MMCV: 1.4.0
MMCV Compiler: GCC 7.3
MMCV CUDA Compiler: 10.2
MMDetection: 2.14.0
MMSegmentation: 0.14.1
MMDetection3D: 0.15.0+04c8e39

@Abyssaledge
Copy link
Collaborator

Could you please set a breakpoint at
https://github.com/TuSimple/SST/blob/main/tools/train.py#L194
then check the content of cfg.data.train?

@chyohoo
Copy link
Author

chyohoo commented Jan 4, 2022

{'type': 'RepeatDataset', 'data_root': '/home/algo_zf/Data/nuscence/', 'ann_file': '/home/algo_zf/Data/nuscence/nuscenes_infos_train.pkl', 'pipeline': [{'type': 'LoadPointsFromFile', 'coord_type': 'LIDAR', 'load_dim': 18, 'use_dim': 18, 'file_client_args': {'backend': 'disk'}}, {'type': 'LoadPointsFromMultiSweeps', 'sweeps_num': 10, 'use_dim': [0, 1, 2, 3, 5, 8, 9], 'pad_empty_sweeps': True, 'file_client_args': {'backend': 'disk'}}, {'type': 'LoadAnnotations3D', 'with_bbox_3d': True, 'with_label_3d': True}, {'type': 'GlobalRotScaleTrans', 'rot_range': [-0.3925, 0.3925], 'scale_ratio_range': [0.95, 1.05], 'translation_std': [0, 0, 0]}, {'type': 'RandomFlip3D', 'flip_ratio_bev_horizontal': 0.5}, {'type': 'PointsRangeFilter', 'point_cloud_range': [0, -50, -5, 50, 50, 3]}, {'type': 'ObjectRangeFilter', 'point_cloud_range': [0, -50, -5, 50, 50, 3]}, {'type': 'ObjectNameFilter', 'classes': ['car', 'truck', 'trailer', 'bus', 'construction_vehicle', 'bicycle', 'motorcycle', 'pedestrian', 'traffic_cone', 'barrier']}, {'type': 'PointShuffle'}, {'type': 'DefaultFormatBundle3D', 'class_names': ['car', 'truck', 'trailer', 'bus', 'construction_vehicle', 'bicycle', 'motorcycle', 'pedestrian', 'traffic_cone', 'barrier']}, {'type': 'Collect3D', 'keys': ['points', 'gt_bboxes_3d', 'gt_labels_3d']}], 'classes': ['car', 'truck', 'trailer', 'bus', 'construction_vehicle', 'bicycle', 'motorcycle', 'pedestrian', 'traffic_cone', 'barrier'], 'modality': {'use_lidar': True, 'use_camera': False, 'use_radar': False, 'use_map': False, 'use_external': False}, 'test_mode': False, 'box_type_3d': 'LiDAR', 'times': 1, 'dataset': {'load_interval': 10}}

@Abyssaledge
Copy link
Collaborator

In cfg.data.train, you have a 'dataset': {'load_interval': 10}, which is wrong. Please refer to https://github.com/TuSimple/SST/blob/main/configs/_base_/datasets/waymo-3d-3class.py#L111
to build the data config.
It would be great if you solve this issue and let me know.

@chyohoo
Copy link
Author

chyohoo commented Jan 4, 2022

hi, changing the load_interval value does nothing helpful to the error mentioned above. But thanks for helping.

@Abyssaledge
Copy link
Collaborator

What I mean is you place the items in wrong place You should fill cfg.train like this:

    train=dict(
        type='RepeatDataset',
        times=2,
        dataset=dict(
            type=dataset_type,
            data_root=data_root,
            ann_file=data_root + 'waymo_infos_train.pkl',
            split='training',
            pipeline=train_pipeline,
            modality=input_modality,
            classes=class_names,
            test_mode=False,
            # we use box_type_3d='LiDAR' in kitti and nuscenes dataset
            # and box_type_3d='Depth' in sunrgbd and scannet dataset.
            box_type_3d='LiDAR',
            load_interval=1)),

data_root, anno_file ... should be placed under cfg.train.dataset instead of cfg.train.

@Abyssaledge
Copy link
Collaborator

@chyohoo Does it work now?

@chyohoo
Copy link
Author

chyohoo commented Jan 4, 2022 •
edited

@chyohoo Does it work now?

thanks! It works!!
But another error occurred.

/opt/conda/conda-bld/pytorch_1639180543123/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [5,0,0], thread: [32,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.

@Abyssaledge
Copy link
Collaborator

Could you please provide more detailed error information with CUDA_LAUNCH_BLOCKING=1 ?

@chyohoo
Copy link
Author

chyohoo commented Jan 4, 2022

2022-01-04 17:48:55,033 - mmdet - INFO - workflow: [('train', 1)], max: 12 epochs
2022-01-04 17:48:55,034 - mmdet - INFO - Checkpoints will be saved to /home/chen_haoye/WS_SST/SST/work_dirs/sst_nuscenes by HardDiskBackend.
drop_info is set to {0: {'max_tokens': 30, 'drop_range': (0, 30)}, 1: {'max_tokens': 60, 'drop_range': (30, 60)}, 2: {'max_tokens': 100, 'drop_range': (60, 100000)}}, in input_layer
/home/chen_haoye/WS_SST/SST/mmdet3d/models/middle_encoders/sst_input_layer.py:321: UserWarning: floordiv is deprecated, and its behavior will change in a future version of pytorch. It currently rounds toward 0 (like the 'trunc' function NOT 'floor'). This results in incorrect rounding for negative values. To keep the current behavior, use torch.div(a, b, rounding_mode='trunc'), or for actual floor division, use torch.div(a, b, rounding_mode='floor').
win_coors_x = shifted_coors_x // win_shape_x
/home/chen_haoye/WS_SST/SST/mmdet3d/models/middle_encoders/sst_input_layer.py:322: UserWarning: floordiv is deprecated, and its behavior will change in a future version of pytorch. It currently rounds toward 0 (like the 'trunc' function NOT 'floor'). This results in incorrect rounding for negative values. To keep the current behavior, use torch.div(a, b, rounding_mode='trunc'), or for actual floor division, use torch.div(a, b, rounding_mode='floor').
win_coors_y = shifted_coors_y // win_shape_y
No voxel belongs to drop_level:1 in shift 0
No voxel belongs to drop_level:2 in shift 0
/home/chen_haoye/WS_SST/SST/mmdet3d/ops/sst/sst_ops.py:50: UserWarning: floordiv is deprecated, and its behavior will change in a future version of pytorch. It currently rounds toward 0 (like the 'trunc' function NOT 'floor'). This results in incorrect rounding for negative values. To keep the current behavior, use torch.div(a, b, rounding_mode='trunc'), or for actual floor division, use torch.div(a, b, rounding_mode='floor').
num_windows = (this_inds // max_tokens).max().item() + 1
/home/chen_haoye/WS_SST/SST/mmdet3d/models/backbones/sst_v1.py:243: UserWarning: floordiv is deprecated, and its behavior will change in a future version of pytorch. It currently rounds toward 0 (like the 'trunc' function NOT 'floor'). This results in incorrect rounding for negative values. To keep the current behavior, use torch.div(a, b, rounding_mode='trunc'), or for actual floor division, use torch.div(a, b, rounding_mode='floor').
inv_freq = self.pos_temperature ** (2 * (inv_freq // 2) / pos_length)
/opt/conda/conda-bld/pytorch_1639180543123/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [69,0,0], thread: [96,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1639180543123/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [69,0,0], thread: [97,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1639180543123/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [69,0,0], thread: [98,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1639180543123/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [69,0,0], thread: [99,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1639180543123/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [69,0,0], thread: [100,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1639180543123/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [69,0,0], thread: [101,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1639180543123/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [69,0,0], thread: [102,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1639180543123/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [69,0,0], thread: [103,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1639180543123/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [69,0,0], thread: [104,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1639180543123/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [69,0,0], thread: [105,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1639180543123/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [69,0,0], thread: [106,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1639180543123/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [69,0,0], thread: [107,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1639180543123/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [69,0,0], thread: [108,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1639180543123/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [69,0,0], thread: [109,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1639180543123/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [69,0,0], thread: [110,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1639180543123/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [69,0,0], thread: [111,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1639180543123/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [69,0,0], thread: [112,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1639180543123/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [69,0,0], thread: [113,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1639180543123/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [69,0,0], thread: [114,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1639180543123/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [69,0,0], thread: [115,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1639180543123/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [69,0,0], thread: [116,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1639180543123/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [69,0,0], thread: [117,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1639180543123/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [69,0,0], thread: [118,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1639180543123/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [69,0,0], thread: [119,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1639180543123/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [69,0,0], thread: [120,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1639180543123/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [69,0,0], thread: [121,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1639180543123/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [69,0,0], thread: [122,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1639180543123/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [69,0,0], thread: [123,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1639180543123/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [69,0,0], thread: [124,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1639180543123/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [69,0,0], thread: [125,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1639180543123/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [69,0,0], thread: [126,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1639180543123/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [69,0,0], thread: [127,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1639180543123/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [69,0,0], thread: [0,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1639180543123/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [69,0,0], thread: [1,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1639180543123/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [69,0,0], thread: [2,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1639180543123/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [69,0,0], thread: [3,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1639180543123/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [69,0,0], thread: [4,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1639180543123/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [69,0,0], thread: [5,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1639180543123/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [69,0,0], thread: [6,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1639180543123/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [69,0,0], thread: [7,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1639180543123/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [69,0,0], thread: [8,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1639180543123/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [69,0,0], thread: [9,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1639180543123/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [69,0,0], thread: [10,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1639180543123/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [69,0,0], thread: [11,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1639180543123/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [69,0,0], thread: [12,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1639180543123/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [69,0,0], thread: [13,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1639180543123/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [69,0,0], thread: [14,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1639180543123/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [69,0,0], thread: [15,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1639180543123/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [69,0,0], thread: [16,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1639180543123/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [69,0,0], thread: [17,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1639180543123/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [69,0,0], thread: [18,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1639180543123/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [69,0,0], thread: [19,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1639180543123/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [69,0,0], thread: [20,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1639180543123/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [69,0,0], thread: [21,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1639180543123/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [69,0,0], thread: [22,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1639180543123/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [69,0,0], thread: [23,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1639180543123/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [69,0,0], thread: [24,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1639180543123/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [69,0,0], thread: [25,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1639180543123/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [69,0,0], thread: [26,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1639180543123/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [69,0,0], thread: [27,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1639180543123/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [69,0,0], thread: [28,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1639180543123/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [69,0,0], thread: [29,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1639180543123/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [69,0,0], thread: [30,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1639180543123/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [69,0,0], thread: [31,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1639180543123/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [69,0,0], thread: [32,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1639180543123/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [69,0,0], thread: [33,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1639180543123/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [69,0,0], thread: [34,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1639180543123/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [69,0,0], thread: [35,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1639180543123/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [69,0,0], thread: [36,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1639180543123/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [69,0,0], thread: [37,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1639180543123/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [69,0,0], thread: [38,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1639180543123/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [69,0,0], thread: [39,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1639180543123/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [69,0,0], thread: [40,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1639180543123/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [69,0,0], thread: [41,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1639180543123/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [69,0,0], thread: [42,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1639180543123/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [69,0,0], thread: [43,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1639180543123/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [69,0,0], thread: [44,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1639180543123/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [69,0,0], thread: [45,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1639180543123/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [69,0,0], thread: [46,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1639180543123/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [69,0,0], thread: [47,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1639180543123/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [69,0,0], thread: [48,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1639180543123/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [69,0,0], thread: [49,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1639180543123/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [69,0,0], thread: [50,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1639180543123/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [69,0,0], thread: [51,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1639180543123/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [69,0,0], thread: [52,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1639180543123/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [69,0,0], thread: [53,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1639180543123/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [69,0,0], thread: [54,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1639180543123/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [69,0,0], thread: [55,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1639180543123/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [69,0,0], thread: [56,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1639180543123/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [69,0,0], thread: [57,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1639180543123/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [69,0,0], thread: [58,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1639180543123/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [69,0,0], thread: [59,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1639180543123/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [69,0,0], thread: [60,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1639180543123/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [69,0,0], thread: [61,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1639180543123/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [69,0,0], thread: [62,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1639180543123/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [69,0,0], thread: [63,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1639180543123/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [69,0,0], thread: [64,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1639180543123/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [69,0,0], thread: [65,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1639180543123/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [69,0,0], thread: [66,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1639180543123/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [69,0,0], thread: [67,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1639180543123/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [69,0,0], thread: [68,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1639180543123/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [69,0,0], thread: [69,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1639180543123/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [69,0,0], thread: [70,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1639180543123/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [69,0,0], thread: [71,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1639180543123/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [69,0,0], thread: [72,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1639180543123/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [69,0,0], thread: [73,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1639180543123/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [69,0,0], thread: [74,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1639180543123/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [69,0,0], thread: [75,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1639180543123/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [69,0,0], thread: [76,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1639180543123/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [69,0,0], thread: [77,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1639180543123/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [69,0,0], thread: [78,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1639180543123/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [69,0,0], thread: [79,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1639180543123/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [69,0,0], thread: [80,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1639180543123/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [69,0,0], thread: [81,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1639180543123/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [69,0,0], thread: [82,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1639180543123/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [69,0,0], thread: [83,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1639180543123/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [69,0,0], thread: [84,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1639180543123/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [69,0,0], thread: [85,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1639180543123/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [69,0,0], thread: [86,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1639180543123/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [69,0,0], thread: [87,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1639180543123/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [69,0,0], thread: [88,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1639180543123/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [69,0,0], thread: [89,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1639180543123/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [69,0,0], thread: [90,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1639180543123/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [69,0,0], thread: [91,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1639180543123/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [69,0,0], thread: [92,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1639180543123/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [69,0,0], thread: [93,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1639180543123/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [69,0,0], thread: [94,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1639180543123/work/aten/src/ATen/native/cuda/IndexKernel.cu:93: operator(): block: [69,0,0], thread: [95,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
Traceback (most recent call last):
File "/home/chen_haoye/WS_SST/SST/tools/train.py", line 234, in
main()
File "/home/chen_haoye/WS_SST/SST/tools/train.py", line 231, in main
meta=meta)
File "/home/chen_haoye/WS_SST/SST/mmdet3d/apis/train.py", line 34, in train_model
meta=meta)
File "/home/chen_haoye/anaconda3/envs/SST/lib/python3.7/site-packages/mmdet/apis/train.py", line 170, in train_detector
runner.run(data_loaders, cfg.workflow)
File "/home/chen_haoye/anaconda3/envs/SST/lib/python3.7/site-packages/mmcv/runner/epoch_based_runner.py", line 127, in run
epoch_runner(data_loaders[i], **kwargs)
File "/home/chen_haoye/anaconda3/envs/SST/lib/python3.7/site-packages/mmcv/runner/epoch_based_runner.py", line 50, in train
self.run_iter(data_batch, train_mode=True, **kwargs)
File "/home/chen_haoye/anaconda3/envs/SST/lib/python3.7/site-packages/mmcv/runner/epoch_based_runner.py", line 30, in run_iter
**kwargs)
File "/home/chen_haoye/anaconda3/envs/SST/lib/python3.7/site-packages/mmcv/parallel/data_parallel.py", line 75, in train_step
return self.module.train_step(*inputs[0], **kwargs[0])
File "/home/chen_haoye/anaconda3/envs/SST/lib/python3.7/site-packages/mmdet/models/detectors/base.py", line 237, in train_step
losses = self(**data)
File "/home/chen_haoye/anaconda3/envs/SST/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
return forward_call(*input, **kwargs)
File "/home/chen_haoye/anaconda3/envs/SST/lib/python3.7/site-packages/mmcv/runner/fp16_utils.py", line 128, in new_func
output = old_func(*new_args, **new_kwargs)
File "/home/chen_haoye/WS_SST/SST/mmdet3d/models/detectors/base.py", line 58, in forward
return self.forward_train(**kwargs)
File "/home/chen_haoye/WS_SST/SST/mmdet3d/models/detectors/voxelnet.py", line 90, in forward_train
x = self.extract_feat(points, img_metas)
File "/home/chen_haoye/WS_SST/SST/mmdet3d/models/detectors/dynamic_voxelnet.py", line 43, in extract_feat
x = self.backbone(x)
File "/home/chen_haoye/anaconda3/envs/SST/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
return forward_call(*input, kwargs)
File "/home/chen_haoye/WS_SST/SST/mmdet3d/models/backbones/sst_v1.py", line 156, in forward
output = self.recover_bev(output, voxel_info['coors'], batch_size)
File "/home/chen_haoye/WS_SST/SST/mmdet3d/models/backbones/sst_v1.py", line 212, in recover_bev
canvas[:, indices] = voxels
RuntimeError: CUDA error: device-side assert triggered
terminate called after throwing an instance of 'c10::CUDAError'
what(): CUDA error: device-side assert triggered
Exception raised from create_event_internal at /opt/conda/conda-bld/pytorch_1639180543123/work/c10/cuda/CUDACachingAllocator.cpp:1211 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x42 (0x7f734de41d62 in /home/chen_haoye/anaconda3/envs/SST/lib/python3.7/site-packages/torch/lib/libc10.so)
frame #1: + 0x1c503 (0x7f734e0a4503 in /home/chen_haoye/anaconda3/envs/SST/lib/python3.7/site-packages/torch/lib/libc10_cuda.so)
frame #2: c10::cuda::CUDACachingAllocator::raw_delete(void) + 0x1a2 (0x7f734e0a4f12 in /home/chen_haoye/anaconda3/envs/SST/lib/python3.7/site-packages/torch/lib/libc10_cuda.so)
frame #3: c10::TensorImpl::release_resources() + 0xa4 (0x7f734de2b314 in /home/chen_haoye/anaconda3/envs/SST/lib/python3.7/site-packages/torch/lib/libc10.so)
frame #4: + 0x294da9 (0x7f71ef23eda9 in /home/chen_haoye/anaconda3/envs/SST/lib/python3.7/site-packages/torch/lib/libtorch_python.so)
frame #5: + 0xae2f29 (0x7f71efa8cf29 in /home/chen_haoye/anaconda3/envs/SST/lib/python3.7/site-packages/torch/lib/libtorch_python.so)
frame #6: THPVariable_subclass_dealloc(_object) + 0x2b9 (0x7f71efa8d249 in /home/chen_haoye/anaconda3/envs/SST/lib/python3.7/site-packages/torch/lib/libtorch_python.so)
frame #7: + 0xe59d2 (0x55a4caf8f9d2 in /home/chen_haoye/anaconda3/envs/SST/bin/python)
frame #8: + 0xe707f (0x55a4caf9107f in /home/chen_haoye/anaconda3/envs/SST/bin/python)
frame #9: + 0xe5dcb (0x55a4caf8fdcb in /home/chen_haoye/anaconda3/envs/SST/bin/python)
frame #10: + 0xe5dcb (0x55a4caf8fdcb in /home/chen_haoye/anaconda3/envs/SST/bin/python)
frame #11: + 0xe5698 (0x55a4caf8f698 in /home/chen_haoye/anaconda3/envs/SST/bin/python)
frame #12: + 0xe6038 (0x55a4caf90038 in /home/chen_haoye/anaconda3/envs/SST/bin/python)
frame #13: + 0xe604e (0x55a4caf9004e in /home/chen_haoye/anaconda3/envs/SST/bin/python)
frame #14: + 0xe604e (0x55a4caf9004e in /home/chen_haoye/anaconda3/envs/SST/bin/python)
frame #15: + 0xe604e (0x55a4caf9004e in /home/chen_haoye/anaconda3/envs/SST/bin/python)
frame #16: + 0xe604e (0x55a4caf9004e in /home/chen_haoye/anaconda3/envs/SST/bin/python)
frame #17: + 0xe604e (0x55a4caf9004e in /home/chen_haoye/anaconda3/envs/SST/bin/python)
frame #18: + 0xe604e (0x55a4caf9004e in /home/chen_haoye/anaconda3/envs/SST/bin/python)
frame #19: + 0xe604e (0x55a4caf9004e in /home/chen_haoye/anaconda3/envs/SST/bin/python)
frame #20: PyDict_SetItem + 0x4bf (0x55a4cafd8b5f in /home/chen_haoye/anaconda3/envs/SST/bin/python)
frame #21: PyDict_SetItemString + 0x66 (0x55a4cafd8e06 in /home/chen_haoye/anaconda3/envs/SST/bin/python)
frame #22: PyImport_Cleanup + 0x9c (0x55a4cb0dd03c in /home/chen_haoye/anaconda3/envs/SST/bin/python)
frame #23: Py_FinalizeEx + 0x67 (0x55a4cb0dd407 in /home/chen_haoye/anaconda3/envs/SST/bin/python)
frame #24: + 0x248b1b (0x55a4cb0f2b1b in /home/chen_haoye/anaconda3/envs/SST/bin/python)
frame #25: _Py_UnixMain + 0x3c (0x55a4cb0f2e9c in /home/chen_haoye/anaconda3/envs/SST/bin/python)
frame #26: __libc_start_main + 0xe7 (0x7f735355bbf7 in /lib/x86_64-linux-gnu/libc.so.6)
frame #27: + 0x1c761d (0x55a4cb07161d in /home/chen_haoye/anaconda3/envs/SST/bin/python)

Aborted (core dumped)

@Abyssaledge
Copy link
Collaborator

According to voxel_size=(0.2, 0.2, 6) and point_cloud_range=[0, -50, -5, 50, 50, 3] in you config, I guess the correct voxel_size you want is (0.2, 0.2, 8) and the output_shape in SSTv1 should be (500, 500).
Please open another issue if you have a different problem.

@chyohoo
Copy link
Author

chyohoo commented Jan 4, 2022

Thanks, after setting the correct output_shape, the error disappeared!

@chyohoo chyohoo mentioned this issue Jan 4, 2022
Closed
@chyohoo chyohoo closed this as completed Jan 5, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@chyohoo @Abyssaledge