RuntimeError: cublas runtime error : the GPU program failed to execute at /pytorch/aten/src/THC/THCBlas.cu:259

[yuxgis]

Hi!
I met some bug when i run test.py. more infomation are list blow:

[2022-06-05 20:51:56,158 INFO scannetv2_inst.py line 97 3304] Testing samples (test): 1
Traceback (most recent call last):
File "test.py", line 269, in
test(model, model_fn, data_name, cfg.test_epoch)
File "test.py", line 57, in test
preds = model_fn(batch, model, epoch)
File "/home/yux/yux/pointCloud/HAIS/model/hais/hais.py", line 371, in test_model_fn
ret = model(input_, p2v_map, coords_float, coords[:, 0].int(), batch_offsets, epoch, 'test')
File "/home/yux/anaconda3/envs/hais/lib/python3.7/site-packages/torch/nn/modules/module.py", line 493, in call
result = self.forward(*input, **kwargs)
File "/home/yux/yux/pointCloud/HAIS/model/hais/hais.py", line 265, in forward
output = self.input_conv(input)
File "/home/yux/anaconda3/envs/hais/lib/python3.7/site-packages/torch/nn/modules/module.py", line 493, in call
result = self.forward(*input, **kwargs)
File "/home/yux/anaconda3/envs/hais/lib/python3.7/site-packages/spconv/modules.py", line 123, in forward
input = module(input)
File "/home/yux/anaconda3/envs/hais/lib/python3.7/site-packages/torch/nn/modules/module.py", line 493, in call
result = self.forward(*input, **kwargs)
File "/home/yux/anaconda3/envs/hais/lib/python3.7/site-packages/spconv/conv.py", line 157, in forward
outids.shape[0])
File "/home/yux/anaconda3/envs/hais/lib/python3.7/site-packages/spconv/functional.py", line 83, in forward
return ops.indice_conv(features, filters, indice_pairs, indice_pair_num, num_activate_out, False, True)
File "/home/yux/anaconda3/envs/hais/lib/python3.7/site-packages/spconv/ops.py", line 112, in indice_conv
int(inverse), int(subm))
RuntimeError: cublas runtime error : the GPU program failed to execute at /pytorch/aten/src/THC/THCBlas.cu:259

pytorch 1.1.0
cuda 10.2
cudnn 8.2.2
spconv 1.0

@outsidercsy