RuntimeError: GET was unable to find an engine to execute this computation

[gw0522]

Hello, I made the following mistake in training. Do you know what the problem is？

python3 main.py --exp_name exp1 --post --emb_off --cmc_off --aw_off --new_kf_off --grid_off --dataset mot17

/home/lab509-2/miniconda3/envs/deepoc/lib/python3.8/site-packages/torch/nn/modules/module.py:831: UserWarning: The .grad attribute of a Tensor that is not a leaf Tensor is being accessed. Its .grad attribute won't be populated during autograd.backward(). If you indeed want the .grad field to be populated for a non-leaf Tensor, use .retain_grad() on the non-leaf Tensor. If you access the non-leaf Tensor by mistake, make sure you access the leaf Tensor instead. See github.com/pytorch/pytorch/pull/30531 for more informations. (Triggered internally at aten/src/ATen/core/TensorBody.h:486.)
if param.grad is not None:
loading annotations into memory...
Done (t=0.12s)
creating index...
index created!

Initializing tracker for MOT17-02-FRCNN
Time spent: 0.000, FPS 0.00

Traceback (most recent call last):
File "main.py", line 193, in
main()
File "main.py", line 135, in main
pred = det(img, tag)
File "/home/lab509-2/miniconda3/envs/deepoc/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/media/lab509-2/6t/gy/Deep-OC-SORT/external/adaptors/detector.py", line 47, in forward
output = self.model(batch)
File "/home/lab509-2/miniconda3/envs/deepoc/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/media/lab509-2/6t/gy/Deep-OC-SORT/external/adaptors/yolox_adaptor.py", line 21, in forward
raw = self.model(batch)
File "/home/lab509-2/miniconda3/envs/deepoc/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/media/lab509-2/6t/gy/Deep-OC-SORT/external/YOLOX/yolox/models/yolox.py", line 30, in forward
fpn_outs = self.backbone(x)
File "/home/lab509-2/miniconda3/envs/deepoc/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/media/lab509-2/6t/gy/Deep-OC-SORT/external/YOLOX/yolox/models/yolo_pafpn.py", line 93, in forward
out_features = self.backbone(input)
File "/home/lab509-2/miniconda3/envs/deepoc/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/media/lab509-2/6t/gy/Deep-OC-SORT/external/YOLOX/yolox/models/darknet.py", line 169, in forward
x = self.stem(x)
File "/home/lab509-2/miniconda3/envs/deepoc/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/media/lab509-2/6t/gy/Deep-OC-SORT/external/YOLOX/yolox/models/network_blocks.py", line 210, in forward
return self.conv(x)
File "/home/lab509-2/miniconda3/envs/deepoc/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/media/lab509-2/6t/gy/Deep-OC-SORT/external/YOLOX/yolox/models/network_blocks.py", line 54, in fuseforward
return self.act(self.conv(x))
File "/home/lab509-2/miniconda3/envs/deepoc/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/home/lab509-2/miniconda3/envs/deepoc/lib/python3.8/site-packages/torch/nn/modules/conv.py", line 463, in forward
return self._conv_forward(input, self.weight, self.bias)
File "/home/lab509-2/miniconda3/envs/deepoc/lib/python3.8/site-packages/torch/nn/modules/conv.py", line 459, in _conv_forward
return F.conv2d(input, weight, bias, self.stride,
RuntimeError: GET was unable to find an engine to execute this computation

[GerardMaggiolino]

This looks like a cuda issue. Maybe reference this: https://stackoverflow.com/questions/75776895/runtimeerror-get-was-unable-to-find-an-engine-to-execute-this-computation-when