[EfficientDet/PyTorch] TypeError: new(): invalid data type 'str' when training EfficientDet on Waymo dataset

[ChongyuNVIDIA]

Related to EfficientDet/PyTorch

Describe the bug
When I try to reproduce the EfficientDet training result on Waymo dataset as described in:
https://github.com/NVIDIA/DeepLearningExamples/tree/master/PyTorch/Detection/Efficientdet
Meet the " TypeError: new(): invalid data type 'str' " issue after loading the Waymo dataset and start training.

To Reproduce
Steps to reproduce the behavior:

1. Git clone 'https://github.com/NVIDIA/DeepLearningExamples', cd DeepLearningExamples/PyTorch/Detection/Efficientdet
2. run 'waymo_tool/waymo_data_converter.py' to downloads and converts the Waymo data into COCO format
3. Change the dataset path according to 'scripts/waymo/train_waymo_AMP_8xA100-80G.sh'
4. Launch './distributed_train.sh 8 /datasets/Waymo_JoC --model efficientdet_d0 -b 8 --amp --lr 0.2 --sync-bn --opt fusedmomentum --warmup-epochs 1 --output Chong_Results/EfficientDet-D0/Waymo/Dense_PT2106_V100/FP16_24E_8N --worker 8 --fill-color mean --model-ema --model-ema-decay 0.999 --eval-after 24 --epochs 24 --save-checkpoint-interval 1 --smoothing 0.0 --waymo --remove-weights class_net box_net anchor --input_size 1536 --num_classes 3 --resume --freeze-layers backbone --waymo-train /datasets/Waymo_JoC/waymo_coco_format_train/images --waymo-val /datasets/Waymo_JoC/waymo_coco_format_val/images --waymo-val-annotation /datasets/Waymo_JoC/waymo_coco_format_val/annotations/annotations.json --waymo-train-annotation /datasets/Waymo_JoC/waymo_coco_format_train/annotations/annotations.json'

Expected behavior
Expect the EfficientDet training on Waymo dataset can be smooth.

Environment

• Container version: pytorch:21.06-py3
• GPUs in the system: 8x Tesla A100-80GB
• CUDA version: 11.4
• CUDA driver version: 470.82.01

The log info for the training execution:
Added key: store_based_barrier_key:1 to store for rank: 6
Added key: store_based_barrier_key:1 to store for rank: 5
Added key: store_based_barrier_key:1 to store for rank: 3
Added key: store_based_barrier_key:1 to store for rank: 2
Added key: store_based_barrier_key:1 to store for rank: 7
Added key: store_based_barrier_key:1 to store for rank: 4
Added key: store_based_barrier_key:1 to store for rank: 1
Added key: store_based_barrier_key:1 to store for rank: 0
Rank 0: Completed store-based barrier for 8 nodes.
Rank 6: Completed store-based barrier for 8 nodes.
Rank 5: Completed store-based barrier for 8 nodes.
Rank 3: Completed store-based barrier for 8 nodes.
Rank 2: Completed store-based barrier for 8 nodes.
Rank 7: Completed store-based barrier for 8 nodes.
Rank 4: Completed store-based barrier for 8 nodes.
Rank 1: Completed store-based barrier for 8 nodes.
Training in distributed mode with multiple processes, 1 GPU per process. Process 2, total 8.
Training in distributed mode with multiple processes, 1 GPU per process. Process 0, total 8.
Training in distributed mode with multiple processes, 1 GPU per process. Process 4, total 8.
Training in distributed mode with multiple processes, 1 GPU per process. Process 1, total 8.
Training in distributed mode with multiple processes, 1 GPU per process. Process 5, total 8.
Training in distributed mode with multiple processes, 1 GPU per process. Process 6, total 8.
Training in distributed mode with multiple processes, 1 GPU per process. Process 3, total 8.
Training in distributed mode with multiple processes, 1 GPU per process. Process 7, total 8.
model does not have attribute module...
model does not have attribute module...
model does not have attribute module...
model does not have attribute module...
model does not have attribute module...
model does not have attribute module...
model does not have attribute module...
model does not have attribute module...
Restoring model state from stete_dict ...
Loaded state_dict from checkpoint './backbone_checkpoints/efficientdet_backbone_efficientnet_b0_pyt_amp_ckpt_21.06.0.pth'
Restoring model state from stete_dict ...
Loaded state_dict from checkpoint './backbone_checkpoints/efficientdet_backbone_efficientnet_b0_pyt_amp_ckpt_21.06.0.pth'
Restoring model state from stete_dict ...
Loaded state_dict from checkpoint './backbone_checkpoints/efficientdet_backbone_efficientnet_b0_pyt_amp_ckpt_21.06.0.pth'
Restoring model state from stete_dict ...
Loaded state_dict from checkpoint './backbone_checkpoints/efficientdet_backbone_efficientnet_b0_pyt_amp_ckpt_21.06.0.pth'
Restoring model state from stete_dict ...
Loaded state_dict from checkpoint './backbone_checkpoints/efficientdet_backbone_efficientnet_b0_pyt_amp_ckpt_21.06.0.pth'
Restoring model state from stete_dict ...
Loaded state_dict from checkpoint './backbone_checkpoints/efficientdet_backbone_efficientnet_b0_pyt_amp_ckpt_21.06.0.pth'
Restoring model state from stete_dict ...
Loaded state_dict from checkpoint './backbone_checkpoints/efficientdet_backbone_efficientnet_b0_pyt_amp_ckpt_21.06.0.pth'
Restoring model state from stete_dict ...
Loaded state_dict from checkpoint './backbone_checkpoints/efficientdet_backbone_efficientnet_b0_pyt_amp_ckpt_21.06.0.pth'
Input size to be passed to dataloaders: 1536
Image size used in model: 1536
DLL 2022-04-06 06:03:54.651781 - PARAMETER model_name : efficientdet_d0 param_count : 3826868
Input size to be passed to dataloaders: 1536
Image size used in model: 1536
Input size to be passed to dataloaders: 1536
Image size used in model: 1536
Input size to be passed to dataloaders: 1536
Image size used in model: 1536
Input size to be passed to dataloaders: 1536
Image size used in model: 1536
Input size to be passed to dataloaders: 1536
Image size used in model: 1536
Input size to be passed to dataloaders: 1536
Image size used in model: 1536
Converted model to use Synchronized BatchNorm. WARNING: You may have issues if using zero initialized BN layers (enabled by default for ResNets) while sync-bn enabled.
Chong_Results/EfficientDet-D0/Waymo/Dense_PT2106_V100/FP16_24E_8N/train does not exist to load checkpoint
Chong_Results/EfficientDet-D0/Waymo/Dense_PT2106_V100/FP16_24E_8N/train does not exist to load checkpoint
Chong_Results/EfficientDet-D0/Waymo/Dense_PT2106_V100/FP16_24E_8N/train does not exist to load checkpoint
Chong_Results/EfficientDet-D0/Waymo/Dense_PT2106_V100/FP16_24E_8N/train does not exist to load checkpoint
Input size to be passed to dataloaders: 1536
Image size used in model: 1536
Chong_Results/EfficientDet-D0/Waymo/Dense_PT2106_V100/FP16_24E_8N/train does not exist to load checkpoint
Chong_Results/EfficientDet-D0/Waymo/Dense_PT2106_V100/FP16_24E_8N/train does not exist to load checkpoint
Chong_Results/EfficientDet-D0/Waymo/Dense_PT2106_V100/FP16_24E_8N/train does not exist to load checkpoint
Chong_Results/EfficientDet-D0/Waymo/Dense_PT2106_V100/FP16_24E_8N/train does not exist to load checkpoint
Using torch DistributedDataParallel. Install NVIDIA Apex for Apex DDP.
DLL 2022-04-06 06:03:56.451268 - PARAMETER Scheduled_epochs : 34
loading annotations into memory...
loading annotations into memory...
loading annotations into memory...
loading annotations into memory...
loading annotations into memory...
loading annotations into memory...
loading annotations into memory...
loading annotations into memory...
Done (t=69.44s)
creating index...
Done (t=71.93s)
creating index...
Done (t=72.49s)
creating index...
Done (t=72.71s)
creating index...
Done (t=72.77s)
creating index...
Done (t=73.04s)
creating index...
Done (t=73.05s)
creating index...
Done (t=73.08s)
creating index...
index created!
index created!
index created!
index created!
index created!
index created!
index created!
index created!
loading annotations into memory...
loading annotations into memory...
loading annotations into memory...
loading annotations into memory...
loading annotations into memory...
loading annotations into memory...
loading annotations into memory...
loading annotations into memory...
Done (t=22.79s)
creating index...
index created!
Done (t=23.14s)
creating index...
Done (t=22.88s)
creating index...
Done (t=23.24s)
creating index...
Done (t=23.59s)
creating index...
Done (t=23.13s)
creating index...
Done (t=23.37s)
creating index...
Done (t=23.39s)
creating index...
index created!
index created!
index created!
index created!
index created!
index created!
index created!
Traceback (most recent call last):
File "train.py", line 635, in
main()
File "train.py", line 461, in main
train_metrics = train_epoch(
File "train.py", line 522, in train_epoch
input, target = next(loader_iter)
File "/ngc_0/Chong_dxxz_Projects/Gitlab/Adversarial_Detection/DeepLearningExamples/Detection/Waymo/data/loader.py", line 84, in iter
for next_input, next_target in self.loader:
File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 521, in next
data = self._next_data()
File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1203, in _next_data
return self._process_data(data)
File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1229, in _process_data
data.reraise()
File "/opt/conda/lib/python3.8/site-packages/torch/_utils.py", line 434, in reraise
raise self.exc_type(msg)
TypeError: Caught TypeError in DataLoader worker process 0.
Original Traceback (most recent call last):
File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/_utils/worker.py", line 287, in _worker_loop
data = fetcher.fetch(index)
File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 47, in fetch
return self.collate_fn(data)
File "/ngc_0/Chong_dxxz_Projects/Gitlab/Adversarial_Detection/DeepLearningExamples/Detection/Waymo/data/loader.py", line 65, in fast_collate
target[tk][i] = torch.tensor(tv, dtype=target[tk].dtype)
TypeError: new(): invalid data type 'str'