RuntimeError: CUBLAS_STATUS_EXECUTION_FAILED at /tmp/pip-install-z25sch0o/minkowskiengine_101b60a34a8c4b73a1c33ae9cb85e161/src/math_functions_gpu.cu:46

[chalth]

Traceback (most recent call last):
File "train.py", line 79, in
main(config)
File "train.py", line 57, in main
trainer.train()
File "/home/lthach/DeepGlobalRegistration-master (1)/core/trainer.py", line 135, in train
self._train_epoch(epoch)
File "/home/lthach/DeepGlobalRegistration-master (1)/core/trainer.py", line 266, in _train_epoch
loss.backward()
File "/home/lthach/anaconda3/envs/dgr/lib/python3.7/site-packages/torch/tensor.py", line 245, in backward
torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
File "/home/lthach/anaconda3/envs/dgr/lib/python3.7/site-packages/torch/autograd/init.py", line 147, in backward
allow_unreachable=True, accumulate_grad=True) # allow_unreachable flag
File "/home/lthach/anaconda3/envs/dgr/lib/python3.7/site-packages/torch/autograd/function.py", line 89, in apply
return self._forward_cls.backward(self, *args) # type: ignore
File "/home/lthach/anaconda3/envs/dgr/lib/python3.7/site-packages/MinkowskiEngine/MinkowskiConvolution.py", line 111, in backward
coordinate_manager._manager,
RuntimeError: CUBLAS_STATUS_EXECUTION_FAILED at /tmp/pip-install-z25sch0o/minkowskiengine_101b60a34a8c4b73a1c33ae9cb85e161/src/math_functions_gpu.cu:46

Hi, when i try to train the DGR model , i meet this problem , do you know how to solve it ? Thanks!