tranning problem

[20210726]

at Step 5: Begin training,here is error:
2023-08-15 19:05:25,999 - mmdet - INFO - workflow: [('train', 1)], max: 24 epochs
INFO:mmdet:workflow: [('train', 1)], max: 24 epochs
2023-08-15 19:05:26.085644: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
Traceback (most recent call last):
File "tools/train.py", line 230, in
main()
File "tools/train.py", line 220, in main
train_model(
File "/waymo/SST/mmdet3d/apis/train.py", line 41, in train_model
train_detector(
File "/root/anaconda3/envs/ctrl/lib/python3.8/site-packages/mmdet/apis/train.py", line 170, in train_detector
runner.run(data_loaders, cfg.workflow)
File "/root/ctrl/mmcv/mmcv/runner/epoch_based_runner.py", line 127, in run
epoch_runner(data_loaders[i], **kwargs)
File "/root/ctrl/mmcv/mmcv/runner/epoch_based_runner.py", line 47, in train
for i, data_batch in enumerate(self.data_loader):
File "/root/anaconda3/envs/ctrl/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 517, in next
data = self._next_data()
File "/root/anaconda3/envs/ctrl/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1199, in _next_data
return self._process_data(data)
File "/root/anaconda3/envs/ctrl/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1225, in _process_data
data.reraise()
File "/root/anaconda3/envs/ctrl/lib/python3.8/site-packages/torch/_utils.py", line 429, in reraise
raise self.exc_type(msg)
KeyError: Caught KeyError in DataLoader worker process 0.
Original Traceback (most recent call last):
File "/root/anaconda3/envs/ctrl/lib/python3.8/site-packages/torch/utils/data/_utils/worker.py", line 202, in _worker_loop
data = fetcher.fetch(index)
File "/root/anaconda3/envs/ctrl/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/root/anaconda3/envs/ctrl/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 44, in
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/root/anaconda3/envs/ctrl/lib/python3.8/site-packages/mmdet/datasets/dataset_wrappers.py", line 151, in getitem
return self.dataset[idx % self._ori_len]
File "/waymo/SST/mmdet3d/datasets/waymo_tracklet_dataset.py", line 284, in getitem
data = self.prepare_train_data(idx)
File "/waymo/SST/mmdet3d/datasets/waymo_tracklet_dataset.py", line 209, in prepare_train_data
input_dict = self.get_data_info(index)
File "/waymo/SST/mmdet3d/datasets/waymo_tracklet_dataset.py", line 139, in get_data_info
trk.set_type(self.cat2id[trk.type_name], 'mmdet3d')
KeyError: 'Pedestrian'

Killing subprocess 2618
Traceback (most recent call last):
File "/root/anaconda3/envs/ctrl/lib/python3.8/runpy.py", line 194, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/root/anaconda3/envs/ctrl/lib/python3.8/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/root/anaconda3/envs/ctrl/lib/python3.8/site-packages/torch/distributed/launch.py", line 340, in
main()
File "/root/anaconda3/envs/ctrl/lib/python3.8/site-packages/torch/distributed/launch.py", line 326, in main
sigkill_handler(signal.SIGTERM, None) # not coming back
File "/root/anaconda3/envs/ctrl/lib/python3.8/site-packages/torch/distributed/launch.py", line 301, in sigkill_handler
raise subprocess.CalledProcessError(returncode=last_return_code, cmd=cmd)
subprocess.CalledProcessError: Command '['/root/anaconda3/envs/ctrl/bin/python3', '-u', 'tools/train.py', '--local_rank=0', 'configs/ctrl/ctrl_veh_24e.py', '--launcher', 'pytorch', '--no-validate']' returned non-zero exit status 1.

I followed the CTRL_instructions.md, use part of waymo-dataset, and ignore step2。There should be a configuration file to fix this problem, but I couldn’t find it.

[Abyssaledge]

The key should not be 'Pedestrian' since you use the vehicle config. I need the command and adopted config for checking.

[20210726]

The key should not be 'Pedestrian' since you use the vehicle config. I need the command and adopted config for checking.

train command is :bash tools/dist_train.sh configs/ctrl/ctrl_veh_24e.py 1 --no-validate

and fsd_base_vehicle.yaml is:

[Abyssaledge]

Why do you use train_gt.bin in the YAML file?

[20210726]

Why do you use train_gt.bin in the YAML file?

i think train_gt.bin is detection result in waymo bin format。so i shoud run （Step 2: Use ImmortalTracker to generate tracking results in training split (bin file format)） first，then use bin file generated in step 2 to train model？

[Abyssaledge]

No, train_gt.bin contains the ground-truth information on training set. What you need here is the proposals on training set.
So do not change bin path to train_gt.bin, only change the split to training and use 'fsd6f6e_vehicle_full_trainset.bin' if you want to generate training data.

[Abyssaledge]

Please reopen this issue if you need further discussion.

[20210726]

Here is my step trying to reproduce CTRL，I want to know is there any wrong ？
especially step2，and Is the config file ‘fsd_base_vehicle.yaml’ correct?

1.prepare waymo data(I only use part of waymo dataset)
1.1 use my python script to generate train.txt val.txt test.txt and idx2timestamp.pkl idx2contextname.pkl
Then cp train.txt val.txt test.txt to ./data/waymo/kitti_format/ImageSets/
cp idx2timestamp.pkl idx2contextname.pkl to ./data/waymo/kitti_format/
1.2 python tools/create_data.py --dataset waymo --root-path ./data/waymo/ --out-dir ./data/waymo/ --workers 128 --extra-tag waymo

Step 1: Generate train_gt.bin once for all. (waymo bin format).
python ./tools/ctrl/generate_train_gt_bin.py
generate file 'train_gt.bin'

python ./tools/ctrl/extract_poses.py
Generate file context2timestamp.pkl and pose.pkl

Step 2: Use ImmortalTracker to generate tracking results in training split (bin file format)
modify file ego_info.py and time_stamp.py like this:

Modify file waymo_convert_detection.sh like this:

then:
bash preparedata/waymo/waymo_preparedata.sh ~/dataset/waymo/waymo_format/
generate files like this :

bash preparedata/waymo/waymo_convert_detection.sh ~/dataset/waymo/waymo_format/train_gt.bin CTRL_FSD_TTA
Generate files like this:
In data/waymo/training/detection/CTRL_FSD_TTA/dets:

Modify file run_mot.sh like this:

Then:
bash run_mot.sh
generate file like this:

Step 3: Generate track input for training
modify file ‘fsd_base_vehicle.yaml’ like this: pred.bin was generated in step 2.

python ./tools/ctrl/generate_track_input.py ./tools/ctrl/data_configs/fsd_base_vehicle.yaml --process 1
generate files like this:

Step 4: Assign candidates GT tracks
python ./tools/ctrl/generate_candidates.py ./tools/ctrl/data_configs/fsd_base_vehicle.yaml --process 1

Step 5: Begin training
bash tools/dist_train.sh configs/ctrl/ctrl_veh_24e.py 1 --no-validate

[20210726]

@Abyssaledge

[JayYangSS]

step 2, I think you shouldn't use train_gt.bin:
bash preparedata/waymo/waymo_convert_detection.sh ~/dataset/waymo/waymo_format/train_gt.bin CTRL_FSD_TTA

you need use base detector to generate prediction result