RuntimeError: cuDNN error: CUDNN_STATUS_NOT_SUPPORTED. This error may appear if you passed in a non-contiguous input

[imransai]

Seems like most of the issues regarding this subject matter is closed. So I am opening this issue again.

Issue description

I am facing this issue when using loss.backward() on my loss function.
Some of the loss function classes I am using are the following.

class CELoss_auxilary(nn.Module):

def __init__(self, auxloss = True):
    
    super(CELoss_auxilary, self).__init__()
    self.logsoftmax = nn.LogSoftmax(dim = 1)                
    self.crossentropyloss = nn.KLDivLoss(reduction = 'batchmean')

def forward(self, pred_auxdc, gt_target, nspatialscales = 16, device = None, params = None):        
    
    if device is None:
        device = torch.device("cpu")        
    nChannels = params['dce_nChannels'] + 2
    gt_resizeddc_target = depth2dc(gt_target, method = 'gaussian3hot', device = device)
    gt_resizeddc_target = gt_resizeddc_target.detach().permute(0, 2, 3, 1).view(-1, nChannels)
    valid_auxmask = (gt_target > 0).detach()
    valid_auxmask = valid_auxmask.view(-1)
    """
    gt_resizeddc_target = gt_resizeddc_target.view(1, nChannels, -1)
    valid_auxmask = (gt_target > 0).detach()
    valid_auxmask = valid_auxmask.view(-1)
    gt_resizeddc_targetsel = gt_resizeddc_target[..., valid_auxmask]
    pred_auxdc = pred_auxdc.view(1, nChannels, -1)
    pred_auxdcsel = pred_auxdc[..., valid_auxmask]
    """
    gt_resizeddc_targetsel = gt_resizeddc_target[valid_auxmask, ...]
    biased_aux = torch.sum(gt_resizeddc_targetsel*torch.log(gt_resizeddc_targetsel + 1e-7), 1)
    
    pred_auxdc = pred_auxdc.permute(0, 2, 3, 1).view(-1, nChannels)
    #valid_auxmask = (torch.sum(gt_resizeddc_target, 1) > 0.0).detach()
    gt_resizeddc_targetsel = gt_resizeddc_target[valid_auxmask, ...]  
    biased_aux = torch.sum(gt_resizeddc_targetsel*torch.log(gt_resizeddc_targetsel + 1e-7), 1)
    pred_auxdcsel = pred_auxdc[valid_auxmask, ...]                
    self.lossaux = self.crossentropyloss(self.logsoftmax(pred_auxdcsel), gt_resizeddc_targetsel)
    #self.lossaux = torch.mean(-torch.sum(gt_resizeddc_targetsel*self.logsoftmax(pred_auxdcsel), 1) + biased_aux)

    return self.lossaux

class MaskedMSELoss(nn.Module):
def init(self):
super(MaskedMSELoss, self).init()

def forward(self, pred, target):
    assert pred.dim() == target.dim(), "inconsistent dimensions"
    valid_mask = (target > 0).detach()
    diff = target - pred
    diff = diff[valid_mask]
    self.loss = (diff**2).mean()        
    return self.loss

I also checked with other loss functions.

System Info

Pytorch 1.4 cuda-toolkit 10.1 ubuntu 16.0.4.

I am facing this problem only during backward computation in training. My evaluation code runs fine.

Interestingly my code runs fine with this combination:
pytorch1.3 cuda-toolkit 10.0
pytorch1.1 cuda-toolkit 9.0

But I need to use the aforementioned combination pytorch 1.4 cuda-toolkit 10.1 for accessing some sparse convolution tools which are only available in CUDA 10.1>= higher. Can anyone help in this regard?

[peterjc123]

Maybe related to #32395.

[imransai]

For now, pytorch 1.4 nightly seems to have solved this problem! Thanks!

[BoltzmannBrain]

Previous suggestions in this thread did not resolve my problem; currently on pytorch-nightly (1.6.0.dev20200525) w/ cuda 10.1.243. Oddly reducing my batch size from 64 to 36 worked 🤷‍♂️

[Hadrien-Cornier]

It happens to me as well with
pytorch=1.5.0
py3.8
cuda10.1.243
cudnn7.6.3_0

It happens when I reach a batch normalization layer with a huge batch size, but when I decrease the batch size the error is gone. It is probably a memory issue that happens when a batch is too big.

[DuckJ]

It happens to me as well with
pytorch=1.2.0
py3
cuda 10.0
It happens when the network forwards the conv layer. I use 6 V100 and horovod to distributed training， batchsize is not huge (128) .
It's strange that this error didn't happen before I expanded the dataset. After I expanded the dataset, the batchsize didn't change, but this error happened

[TangDL]

maybe input size too large

[ShoufaChen]

Just reduce the batch size and try again. It works for me.

[yarkable]

Thanks, reducing the batch_size is OK. But it seems a bug?

[leopd]

I'm getting this with pytorch 1.6.0. Definitely seems like a bug. Reducing batch size is not a good workaround. Getting the right batch size is critical to certain algorithms & loss functions, such as when doing negative sampling for contrastive learning.

Perhaps related, sometimes my code hits this error instead THCudaTensor sizes too large for THCDeviceTensor conversion which is logged here #24401

[CoderHHX]

It also happens to me
pytorch=1.4.0
py3
cuda 10.1.
Reduce the batch size was not working.
Finally, I change pytorch to 1.7 cuda 11.0, it works. lol~~~

[absorbguo]

reduce batch works for me

[nadir121]

Why is this closed? Reducing the batch size does not solve it for me. It is still a bug.

[tdchua]

I was experiencing this problem as well from pytorch=1.4.0... upgrading to pytorch=1.5 solved the issue 😄

[rongduo]

Just reduce the batch size and try again. It works for me.

Thanks for your suggestion. It resolves my issue.

[ooodragon94]

Why is this closed? Reducing the batch size does not solve it for me. It is still a bug.

totally agreed. We ALWAYS need more batches

[Sanqiang]

It looks like OOM causes this problem too

[meisa233]

add this line after import torch

torch.backends.cudnn.enabled = False

[Sil3ntKn1ght]

any solution for Python 3.10 +
so slow without it.