CUDA Error while evaluation on KITTI dataset:

[SaShukla090]

CUDA Error while evaluation on KITTI dataset:

2023-08-12 12:11:03,009 INFO ---- EPOCH 44 JOINT EVALUATION ----
2023-08-12 12:11:03,009 INFO ==> Output file: ./epnet_plus_plus_released_trained_models/PED/eval_results/eval/epoch_44/val
eval: 0%| | 0/3769 [00:00<?, ?it/s]Traceback (most recent call last):
File "eval_rcnn.py", line 1026, in
eval_single_ckpt(root_result_dir, data_path=args.data_path)
File "eval_rcnn.py", line 868, in eval_single_ckpt
eval_one_epoch(model, test_loader, epoch_id, root_result_dir, logger)
File "eval_rcnn.py", line 790, in eval_one_epoch
ret_dict = eval_one_epoch_joint(model, dataloader, epoch_id, result_dir, logger)
File "eval_rcnn.py", line 555, in eval_one_epoch_joint
ret_dict = model(input_data)
File "/home/truartadmin/anaconda3/envs/3DOD/lib/python3.7/site-packages/torch/nn/modules/module.py", line 547, in call
result = self.forward(*input, **kwargs)
File "/home/truartadmin/Shivam_Shukla/SHR/CyDAS_Object_Detection/3D_OD/EPNetV2/tools/../lib/net/point_rcnn.py", line 53, in forward
rpn_output = self.rpn(input_data)
File "/home/truartadmin/anaconda3/envs/3DOD/lib/python3.7/site-packages/torch/nn/modules/module.py", line 547, in call
result = self.forward(*input, **kwargs)
File "/home/truartadmin/Shivam_Shukla/SHR/CyDAS_Object_Detection/3D_OD/EPNetV2/tools/../lib/net/rpn.py", line 126, in forward
backbone_xyz, backbone_features, img_feature, l_xy_cor = self.backbone_net(pts_input, img_input, xy_input) # (B, N, 3), (B, C, N)
File "/home/truartadmin/anaconda3/envs/3DOD/lib/python3.7/site-packages/torch/nn/modules/module.py", line 547, in call
result = self.forward(*input, **kwargs)
File "/home/truartadmin/Shivam_Shukla/SHR/CyDAS_Object_Detection/3D_OD/EPNetV2/tools/../lib/net/pointnet2_msg.py", line 341, in forward
image = self.Cross_Fusion[i](li_features, first_img_gather_feature, li_xy_cor, image)
File "/home/truartadmin/anaconda3/envs/3DOD/lib/python3.7/site-packages/torch/nn/modules/module.py", line 547, in call
result = self.forward(*input, **kwargs)
File "/home/truartadmin/Shivam_Shukla/SHR/CyDAS_Object_Detection/3D_OD/EPNetV2/tools/../lib/net/pointnet2_msg.py", line 116, in forward
point_features = self.P2IA_Layer(img_features, point_features)
File "/home/truartadmin/anaconda3/envs/3DOD/lib/python3.7/site-packages/torch/nn/modules/module.py", line 547, in call
result = self.forward(*input, **kwargs)
File "/home/truartadmin/Shivam_Shukla/SHR/CyDAS_Object_Detection/3D_OD/EPNetV2/tools/../lib/net/pointnet2_msg.py", line 91, in forward
ri = self.fc1(img_feas_f)
File "/home/truartadmin/anaconda3/envs/3DOD/lib/python3.7/site-packages/torch/nn/modules/module.py", line 547, in call
result = self.forward(*input, **kwargs)
File "/home/truartadmin/anaconda3/envs/3DOD/lib/python3.7/site-packages/torch/nn/modules/linear.py", line 87, in forward
return F.linear(input, self.weight, self.bias)
File "/home/truartadmin/anaconda3/envs/3DOD/lib/python3.7/site-packages/torch/nn/functional.py", line 1369, in linear
ret = torch.addmm(bias, input, weight.t())
RuntimeError: CUDA error: CUBLAS_STATUS_EXECUTION_FAILED when calling cublasSgemm( handle, opa, opb, m, n, k, &alpha, a, lda, b, ldb, &beta, c, ldc)
eval: 0%|