cupy.cuda.cudnn.CuDNNError in NStepLSTM #1803

Closed
zacapa opened this Issue Oct 21, 2016 · 7 comments

Projects

None yet

3 participants

@zacapa
zacapa commented Oct 21, 2016 edited

I'm trying to use NStepLSTM like as follows. But if n_layer is more than 2 (n_layer>=2), I get error 'CUDNN_STATUS_BAD_PARAM'. When n_layer==1, my code make success. Is there anything wrong in my code?

Source code is here.

class RNN(chainer.Chain):

    def __init__(self, n_layer, n_vocab, n_units, dropout, cudnn):
        super(RNN, self).__init__(
            embed=L.EmbedID(n_vocab, n_units),
            l1=L.NStepLSTM(n_layer, n_units, n_units,
                           dropout, use_cudnn=cudnn),
            l2=L.Linear(n_units, n_vocab),
        )

    def __call__(self, hx, cx, xs, train=True):
        xs = [self.embed(item) for item in xs]
        hy, cy, ys = self.l1(hx, cx, xs, train=train)
        y = [self.l2(item) for item in ys]
        return y
$ python ptb_nslstm.py -g 0 -l 2 --cudnn
File "/anaconda3/lib/python3.5/site-packages/chainer/links/connection/n_step_lstm.py", line 96, in __call__
    train=train, use_cudnn=self.use_cudnn)
  File "/anaconda3/lib/python3.5/site-packages/chainer/functions/connection/n_step_lstm.py", line 468, in n_step_lstm
    ret = rnn(*inputs)
  File "/anaconda3/lib/python3.5/site-packages/chainer/function.py", line 197, in __call__
    outputs = self.forward(in_data)
  File "/anaconda3/lib/python3.5/site-packages/chainer/functions/connection/n_step_lstm.py", line 267, in forward
    self.reserve_space.data.ptr, reserve_size)
  File "cupy/cuda/cudnn.pyx", line 983, in cupy.cuda.cudnn.RNNForwardTraining (cupy/cuda/cudnn.cpp:13268)
  File "cupy/cuda/cudnn.pyx", line 1002, in cupy.cuda.cudnn.RNNForwardTraining (cupy/cuda/cudnn.cpp:13008)
  File "cupy/cuda/cudnn.pyx", line 321, in cupy.cuda.cudnn.check_status (cupy/cuda/cudnn.cpp:1718)
cupy.cuda.cudnn.CuDNNError: CUDNN_STATUS_BAD_PARAM: b'CUDNN_STATUS_BAD_PARAM'
@unnonouno
Contributor

Thank you for reporting a problem. Which version of CUDA and cuDNN do you use?

@zacapa
zacapa commented Oct 21, 2016 edited

I have 2 GPUPC. Both of GPUPC have same error.

There are version of CUDA and cuDNN in GPUPC1.

  • CUDA: 8.0
$ nvcc -V
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2016 NVIDIA Corporation
Built on Wed_May__4_21:01:56_CDT_2016
Cuda compilation tools, release 8.0, V8.0.26
  • cuDNN: 5.1
$ python
Python 3.5.2 |Anaconda 4.1.1 (64-bit)| (default, Jul  2 2016, 17:53:06)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import chainer.cuda as cuda
>>> cuda.cudnn.cudnn.getVersion()
5103

There are version of CUDA and cuDNN in GPUPC2.

  • CUDA: 7.5
$ nvcc -V
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2015 NVIDIA Corporation
Built on Tue_Aug_11_14:27:32_CDT_2015
Cuda compilation tools, release 7.5, V7.5.17 
  • cuDNN: 5.1
$ python
Python 3.5.2 |Anaconda 4.1.1 (64-bit)| (default, Jul  2 2016, 17:53:06)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import chainer.cuda as cuda
>>> cuda.cudnn.cudnn.getVersion()
5103
@unnonouno
Contributor

Thank you. I can reproduce this error. I'll fix it. Please wait.

@unnonouno unnonouno added the bug label Oct 21, 2016
@unnonouno
Contributor

I tried to fix the problem. Can you test #1804?

@zacapa
zacapa commented Oct 21, 2016 edited

I tried #1804. The problem is solved. Thank you for supporting.

@unnonouno
Contributor

OK. I'll ask nvidia if this fix is correct.

@unnonouno
Contributor

I confirmed that this fix is correct.

@okuta okuta closed this in #1804 Oct 24, 2016
@delta2323 delta2323 added this to the 1.18.0 milestone Nov 14, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment