RuntimeError: cuDNN error: CUDNN_STATUS_NOT_SUPPORTED. This error may appear if you passed in a non-contiguous input.

[raotnameh]

🐛 Bug

I am using nn.ctc_loss and this is the error I get when I use torch.backends.cudnn.enabled = True.
If I disable the cuddn backends the error goes away.

File "train.py", line 293, in
asr_loss = torch.mean(models['predictor'][1](asr_out.contiguous().log_softmax(2).float(), targets, asr_out_sizes.cpu(), target_sizes)) # average the loss by minibatch
File "/home/hemant/.conda/envs/deep/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
result = self.forward(*input, **kwargs)
File "/home/hemant/.conda/envs/deep/lib/python3.8/site-packages/torch/nn/modules/loss.py", line 1501, in forward
return F.ctc_loss(log_probs, targets, input_lengths, target_lengths, self.blank, self.reduction,
File "/home/hemant/.conda/envs/deep/lib/python3.8/site-packages/torch/nn/functional.py", line 2200, in ctc_loss
return torch.ctc_loss(log_probs, targets, input_lengths, target_lengths, blank, _Reduction.get_enum(reduction),
RuntimeError: cuDNN error: CUDNN_STATUS_NOT_SUPPORTED. This error may appear if you passed in a non-contiguous input.

Environment info produced using: https://github.com/pytorch/pytorch/blob/master/torch/utils/collect_env.py

Collecting environment information...
PyTorch version: 1.7.1
Is debug build: False
CUDA used to build PyTorch: 11.0
ROCM used to build PyTorch: N/A

OS: Ubuntu 18.04.4 LTS (x86_64)
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
Clang version: Could not collect
CMake version: version 3.10.2

Python version: 3.8 (64-bit runtime)
Is CUDA available: True
CUDA runtime version: 11.0.221
GPU models and configuration:
GPU 0: GeForce RTX 3090
GPU 1: GeForce RTX 2080 Ti
GPU 2: GeForce RTX 2080 Ti
GPU 3: GeForce GTX 1080 Ti
GPU 4: GeForce RTX 2080 Ti
GPU 5: GeForce RTX 2080 Ti
GPU 6: GeForce RTX 2080 Ti
GPU 7: GeForce RTX 3090

Nvidia driver version: 460.32.03
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.7.6.5
/usr/local/cuda-11.0/targets/x86_64-linux/lib/libcudnn.so.8
/usr/local/cuda-11.0/targets/x86_64-linux/lib/libcudnn_adv_infer.so.8
/usr/local/cuda-11.0/targets/x86_64-linux/lib/libcudnn_adv_train.so.8
/usr/local/cuda-11.0/targets/x86_64-linux/lib/libcudnn_cnn_infer.so.8
/usr/local/cuda-11.0/targets/x86_64-linux/lib/libcudnn_cnn_train.so.8
/usr/local/cuda-11.0/targets/x86_64-linux/lib/libcudnn_ops_infer.so.8
/usr/local/cuda-11.0/targets/x86_64-linux/lib/libcudnn_ops_train.so.8
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] numpy==1.19.2
[pip3] torch==1.7.1
[pip3] torchaudio==0.7.0a0+a853dff
[pip3] torchfile==0.1.0
[pip3] torchvision==0.8.2
[conda] blas 1.0 mkl
[conda] cudatoolkit 11.0.221 h6bb024c_0
[conda] mkl 2020.2 256
[conda] mkl-service 2.3.0 py38he904b0f_0
[conda] mkl_fft 1.3.0 py38h54f3939_0
[conda] mkl_random 1.1.1 py38h0573a6f_0
[conda] numpy 1.19.2 py38h54aff64_0
[conda] numpy-base 1.19.2 py38hfa32c7d_0
[conda] pytorch 1.7.1 py3.8_cuda11.0.221_cudnn8.0.5_0 pytorch
[conda] torchaudio 0.7.2 py38 pytorch
[conda] torchfile 0.1.0 pypi_0 pypi
[conda] torchvision 0.8.2 py38_cu110 pytorch

cc @csarofeen @ptrblck @xwang233

[raotnameh]

@t-vi

[t-vi]

Thank you for reporting! Seems like we need to add a contiguous call. I'll send a fix. As a workaround you might call contiguous() on your inputs to ctc_loss.

[raotnameh]

@t-vi I do not get this error for all the samples. I get this error after a certain number of iterations depending on the batch size. If the batch size is 32 I get this error at the 1478 iteration and if the batch size is 16 I get the error at the 2 X 1478 iteration.
It means the error comes only if the dataset is large.

Any thoughts on why is this happening?

Furhtermore, I do call .contiguous() on all the inputs to the nn.ctc_loss().

asr_loss = ctc_loss(asr_out.log_softmax(2).contiguous(), targets.contiguous(), asr_out_sizes.contiguous(), target_sizes.contiguous()) 

[t-vi]

Ah, right. Can you save the inputs that cause it?

[raotnameh]

@t-vi Hi, thanks for the quick replies.

Here's the link to the folder from where you can find the saved inputs to the ctc_loss func: Link

So, I am getting a -inf value at one index in the asr_out when using cudnn True. This is not true just for one batch but multiple batches.

A quick fix would be to check if the below is valid or no. If not then return zero error. OR to check for inf values.

asr_out.sum()

Hope you can look into it why the discrepancy. The dataset I use is Librispeech. Let me know if you need any additional details.

[gchanan]

can you please fill out the https://github.com/pytorch/pytorch/blob/master/torch/utils/collect_env.py information?

[t-vi]

I can reproduce this.

N = 1
S = 256
C = 10
T = 500
target = torch.randint(low=1, high=C, size=(S,), dtype=torch.int)
input_lengths = torch.full(size=(N,), fill_value=T, dtype=torch.int)
target_lengths = torch.tensor(S, dtype=torch.int)
inp = torch.randn(T, N, C, device='cuda').log_softmax(2).requires_grad_()
print(torch.backends.cudnn.version())
torch.nn.functional.ctc_loss(inp, target, input_lengths, target_lengths, reduction='none')

From the NVidia docs:

CUDNN_STATUS_BAD_PARAM

    At least one of the following conditions are met:

        The dimensions of probsDesc do not match the dimensions of gradientsDesc.
        The inputLengths do not agree with the first dimension of probsDesc.
        The workSpaceSizeInBytes is not sufficient.
        The labelLengths is greater than 256.

CUDNN_STATUS_NOT_SUPPORTED

    A compute or data type other than FLOAT was chosen, or an unknown algorithm type was chosen.

probably they meant to add

CUDNN_STATUS_NOT_SUPPORTED
        The labelLengths is 256.

(On a more serious note, I would speculate that the invalid param test checks what is documented but for some reason - either because it indeed is a problem or erroneously - a later check complains)

We don't call cudnn with > 256, maybe we should avoid it with 256, too.

@ptrblck

[t-vi]

I'll send a PR to take 256 length inputs to the native backend.

[raotnameh]

can you please fill out the https://github.com/pytorch/pytorch/blob/master/torch/utils/collect_env.py information?

Done. Added the info to the issue itself:

FYI

Environment info produced using: https://github.com/pytorch/pytorch/blob/master/torch/utils/collect_env.py

Collecting environment information...
PyTorch version: 1.7.1
Is debug build: False
CUDA used to build PyTorch: 11.0
ROCM used to build PyTorch: N/A

OS: Ubuntu 18.04.4 LTS (x86_64)
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
Clang version: Could not collect
CMake version: version 3.10.2

Python version: 3.8 (64-bit runtime)
Is CUDA available: True
CUDA runtime version: 11.0.221
GPU models and configuration:
GPU 0: GeForce RTX 3090
GPU 1: GeForce RTX 2080 Ti
GPU 2: GeForce RTX 2080 Ti
GPU 3: GeForce GTX 1080 Ti
GPU 4: GeForce RTX 2080 Ti
GPU 5: GeForce RTX 2080 Ti
GPU 6: GeForce RTX 2080 Ti
GPU 7: GeForce RTX 3090

Nvidia driver version: 460.32.03
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.7.6.5
/usr/local/cuda-11.0/targets/x86_64-linux/lib/libcudnn.so.8
/usr/local/cuda-11.0/targets/x86_64-linux/lib/libcudnn_adv_infer.so.8
/usr/local/cuda-11.0/targets/x86_64-linux/lib/libcudnn_adv_train.so.8
/usr/local/cuda-11.0/targets/x86_64-linux/lib/libcudnn_cnn_infer.so.8
/usr/local/cuda-11.0/targets/x86_64-linux/lib/libcudnn_cnn_train.so.8
/usr/local/cuda-11.0/targets/x86_64-linux/lib/libcudnn_ops_infer.so.8
/usr/local/cuda-11.0/targets/x86_64-linux/lib/libcudnn_ops_train.so.8
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] numpy==1.19.2
[pip3] torch==1.7.1
[pip3] torchaudio==0.7.0a0+a853dff
[pip3] torchfile==0.1.0
[pip3] torchvision==0.8.2
[conda] blas 1.0 mkl
[conda] cudatoolkit 11.0.221 h6bb024c_0
[conda] mkl 2020.2 256
[conda] mkl-service 2.3.0 py38he904b0f_0
[conda] mkl_fft 1.3.0 py38h54f3939_0
[conda] mkl_random 1.1.1 py38h0573a6f_0
[conda] numpy 1.19.2 py38h54aff64_0
[conda] numpy-base 1.19.2 py38hfa32c7d_0
[conda] pytorch 1.7.1 py3.8_cuda11.0.221_cudnn8.0.5_0 pytorch
[conda] torchaudio 0.7.2 py38 pytorch
[conda] torchfile 0.1.0 pypi_0 pypi
[conda] torchvision 0.8.2 py38_cu110 pytorch