CUDA error

[alsheabi]

`100%
333/333 [01:51<00:00, 3.44it/s]
Caught RuntimeError in replica 0 on device 0.
Original Traceback (most recent call last):
File "/opt/conda/envs/efdet/lib/python3.7/site-packages/torch/nn/parallel/parallel_apply.py", line 60, in _worker
output = module(*input, **kwargs)
File "/opt/conda/envs/efdet/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in call
result = self.forward(*input, **kwargs)
File "Monk_Object_Detection/4_efficientdet/lib/src/model.py", line 304, in forward
c3, c4, c5 = self.backbone_net(img_batch)
File "/opt/conda/envs/efdet/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in call
result = self.forward(*input, **kwargs)
File "Monk_Object_Detection/4_efficientdet/lib/src/model.py", line 187, in forward
x = self.model._swish(self.model._bn0(self.model._conv_stem(x)))
File "/opt/conda/envs/efdet/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in call
result = self.forward(*input, **kwargs)
File "/opt/conda/envs/efdet/lib/python3.7/site-packages/efficientnet_pytorch/utils.py", line 144, in forward
x = F.conv2d(x, self.weight, self.bias, self.stride, self.padding, self.dilation, self.groups)
RuntimeError: cuDNN error: CUDNN_STATUS_NOT_INITIALIZED

CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
CUDA error: the launch timed out and was terminated
/opt/conda/envs/efdet/lib/python3.7/site-packages/numpy/core/fromnumeric.py:3373: RuntimeWarning: Mean of empty slice.
out=out, **kwargs)
/opt/conda/envs/efdet/lib/python3.7/site-packages/numpy/core/_methods.py:170: RuntimeWarning: invalid value encountered in double_scalars
ret = ret.dtype.type(ret / rcount)

RuntimeError Traceback (most recent call last)
in
1 #%%time
----> 2 gtf.Train(num_epochs=50, model_output_dir="trained/")

~/DS/Monk_Object_Detection/4_efficientdet/lib/train_detector.py in Train(self, num_epochs, model_output_dir)
357 with torch.no_grad():
358 if torch.cuda.is_available():
--> 359 cls_loss, reg_loss = self.system_dict["local"]["model"]([data['img'].to(self.system_dict["local"]["device"]).float(), data['annot'].to(self.system_dict["local"]["device"])])
360 else:
361 cls_loss, reg_loss = self.system_dict["local"]["model"]([data['img'].float(), data['annot']])

RuntimeError: CUDA error: the launch timed out and was terminated`