New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

failed to install chainer with cuda 8.0 / cuDNN v6 #2501

Closed
furushchev opened this Issue Apr 4, 2017 · 10 comments

Comments

Projects
None yet
5 participants
@furushchev

furushchev commented Apr 4, 2017

Hi,

I installed CUDA Toolkit 8.0 and cuDNN v6.
I then tried to install chainer, but it failed with errors as below:

    cupy/cuda/cudnn.cpp: In function ‘PyObject* __pyx_f_4cupy_4cuda_5cudnn_setFilterNdDescriptor_v3(size_t, int, int, size_t, int)’:
    cupy/cuda/cudnn.cpp:4203:177: error: ‘cudnnSetFilterNdDescriptor_v3’ was not declared in this scope
       __pyx_v_status = cudnnSetFilterNdDescriptor_v3(((cudnnFilterDescriptor_t)__pyx_v_filterDesc), ((cudnnDataType_t)__pyx_v_dataType), __pyx_v_nbDims, ((int *)__pyx_v_filterDimA));
                                                                                                                                                                                     ^
    cupy/cuda/cudnn.cpp: In function ‘PyObject* __pyx_f_4cupy_4cuda_5cudnn_setFilter4dDescriptor_v4(size_t, int, int, int, int, int, int, int)’:
    cupy/cuda/cudnn.cpp:4359:215: error: ‘cudnnSetFilter4dDescriptor_v4’ was not declared in this scope
       __pyx_v_status = cudnnSetFilter4dDescriptor_v4(((cudnnFilterDescriptor_t)__pyx_v_filterDesc), ((cudnnDataType_t)__pyx_v_dataType), ((cudnnTensorFormat_t)__pyx_v_format), __pyx_v_k, __pyx_v_c, __pyx_v_h, __pyx_v_w);
                                                                                                                                                                                                                           ^
    cupy/cuda/cudnn.cpp: In function ‘PyObject* __pyx_f_4cupy_4cuda_5cudnn_setFilterNdDescriptor_v4(size_t, int, int, int, size_t, int)’:
    cupy/cuda/cudnn.cpp:4542:216: error: ‘cudnnSetFilterNdDescriptor_v4’ was not declared in this scope
       __pyx_v_status = cudnnSetFilterNdDescriptor_v4(((cudnnFilterDescriptor_t)__pyx_v_filterDesc), ((cudnnDataType_t)__pyx_v_dataType), ((cudnnTensorFormat_t)__pyx_v_format), __pyx_v_nbDims, ((int *)__pyx_v_filterDimA));
                                                                                                                                                                                                                            ^
    cupy/cuda/cudnn.cpp: In function ‘PyObject* __pyx_f_4cupy_4cuda_5cudnn_getFilterNdDescriptor(size_t, int, int)’:
    cupy/cuda/cudnn.cpp:4729:202: error: ‘cudnnGetFilterNdDescriptor_v4’ was not declared in this scope
       __pyx_v_status = cudnnGetFilterNdDescriptor_v4(((cudnnFilterDescriptor_t)__pyx_v_wDesc), __pyx_v_nbDimsRequested, (&__pyx_v_dataType), (&__pyx_v_format), (&__pyx_v_nbDims), (&(__pyx_v_filterDimA[0])));
                                                                                                                                                                                                              ^
    cupy/cuda/cudnn.cpp: In function ‘PyObject* __pyx_f_4cupy_4cuda_5cudnn_setConvolution2dDescriptor_v4(size_t, int, int, int, int, int, int, int, int)’:
    cupy/cuda/cudnn.cpp:5112:231: error: too few arguments to function ‘cudnnStatus_t cudnnSetConvolution2dDescriptor(cudnnConvolutionDescriptor_t, int, int, int, int, int, int, cudnnConvolutionMode_t, cudnnDataType_t)’
       __pyx_v_status = cudnnSetConvolution2dDescriptor_v4(((cudnnConvolutionDescriptor_t)__pyx_v_convDesc), __pyx_v_pad_h, __pyx_v_pad_w, __pyx_v_u, __pyx_v_v, __pyx_v_upscalex, __pyx_v_upscaley, ((cudnnConvolutionMode_t)__pyx_v_mode));
                                                                                                                                                                                                                                           ^
    In file included from cupy/cuda/cupy_cudnn.h:10:0,
                     from cupy/cuda/cudnn.cpp:481:
    /usr/local/cuda/include/cudnn.h:500:27: note: declared here
     cudnnStatus_t CUDNNWINAPI cudnnSetConvolution2dDescriptor( cudnnConvolutionDescriptor_t convDesc,
                               ^
    cupy/cuda/cudnn.cpp: In function ‘PyObject* __pyx_f_4cupy_4cuda_5cudnn_setPooling2dDescriptor_v3(size_t, int, int, int, int, int, int, int, int)’:
    cupy/cuda/cudnn.cpp:10226:279: error: ‘cudnnSetPooling2dDescriptor_v3’ was not declared in this scope
       __pyx_v_status = cudnnSetPooling2dDescriptor_v3(((cudnnPoolingDescriptor_t)__pyx_v_poolingDesc), ((cudnnPoolingMode_t)__pyx_v_mode), __pyx_v_windowHeight, __pyx_v_windowWidth, __pyx_v_verticalPadding, __pyx_v_horizontalPadding, __pyx_v_verticalStride, __pyx_v_horizontalStride);
                                                                                                                                                                                                                                                                                           ^
    cupy/cuda/cudnn.cpp: In function ‘PyObject* __pyx_f_4cupy_4cuda_5cudnn_setPooling2dDescriptor_v4(size_t, int, int, int, int, int, int, int, int, int)’:
    cupy/cuda/cudnn.cpp:10418:330: error: ‘cudnnSetPooling2dDescriptor_v4’ was not declared in this scope
       __pyx_v_status = cudnnSetPooling2dDescriptor_v4(((cudnnPoolingDescriptor_t)__pyx_v_poolingDesc), ((cudnnPoolingMode_t)__pyx_v_mode), ((cudnnNanPropagation_t)__pyx_v_maxpoolingNanOpt), __pyx_v_windowHeight, __pyx_v_windowWidth, __pyx_v_verticalPadding, __pyx_v_horizontalPadding, __pyx_v_verticalStride, __pyx_v_horizontalStride);
                                                                                                                                                                                                                                                                                                                                              ^
    cupy/cuda/cudnn.cpp: In function ‘PyObject* __pyx_f_4cupy_4cuda_5cudnn_setPoolingNdDescriptor_v3(size_t, int, int, size_t, size_t, size_t, int)’:
    cupy/cuda/cudnn.cpp:10619:232: error: ‘cudnnSetPoolingNdDescriptor_v3’ was not declared in this scope
       __pyx_v_status = cudnnSetPoolingNdDescriptor_v3(((cudnnPoolingDescriptor_t)__pyx_v_poolingDesc), ((cudnnPoolingMode_t)__pyx_v_mode), __pyx_v_nbDims, ((int *)__pyx_v_windowDimA), ((int *)__pyx_v_paddingA), ((int *)__pyx_v_strideA));
                                                                                                                                                                                                                                            ^
    cupy/cuda/cudnn.cpp: In function ‘PyObject* __pyx_f_4cupy_4cuda_5cudnn_setPoolingNdDescriptor_v4(size_t, int, int, int, size_t, size_t, size_t, int)’:
    cupy/cuda/cudnn.cpp:10793:283: error: ‘cudnnSetPoolingNdDescriptor_v4’ was not declared in this scope
       __pyx_v_status = cudnnSetPoolingNdDescriptor_v4(((cudnnPoolingDescriptor_t)__pyx_v_poolingDesc), ((cudnnPoolingMode_t)__pyx_v_mode), ((cudnnNanPropagation_t)__pyx_v_maxpoolingNanOpt), __pyx_v_nbDims, ((int *)__pyx_v_windowDimA), ((int *)__pyx_v_paddingA), ((int *)__pyx_v_strideA));
                                                                                                                                                                                                                                                                                               ^
    cupy/cuda/cudnn.cpp: In function ‘PyObject* __pyx_f_4cupy_4cuda_5cudnn_activationForward_v3(size_t, int, size_t, size_t, size_t, size_t, size_t, size_t, int)’:
    cupy/cuda/cudnn.cpp:13483:313: error: ‘cudnnActivationForward_v3’ was not declared in this scope
             __pyx_v_status = cudnnActivationForward_v3(((cudnnHandle_t)__pyx_v_handle), ((cudnnActivationMode_t)__pyx_v_mode), ((void *)__pyx_v_alpha), ((cudnnTensorDescriptor_t)__pyx_v_srcDesc), ((void *)__pyx_v_srcData), ((void *)__pyx_v_beta), ((cudnnTensorDescriptor_t)__pyx_v_dstDesc), ((void *)__pyx_v_dstData));
                                                                                                                                                                                                                                                                                                                             ^
    cupy/cuda/cudnn.cpp: In function ‘PyObject* __pyx_f_4cupy_4cuda_5cudnn_activationForward_v4(size_t, size_t, size_t, size_t, size_t, size_t, size_t, size_t, int)’:
    cupy/cuda/cudnn.cpp:13708:329: error: ‘cudnnActivationForward_v4’ was not declared in this scope
             __pyx_v_status = cudnnActivationForward_v4(((cudnnHandle_t)__pyx_v_handle), ((cudnnActivationDescriptor_t)__pyx_v_activationDesc), ((void *)__pyx_v_alpha), ((cudnnTensorDescriptor_t)__pyx_v_srcDesc), ((void *)__pyx_v_srcData), ((void *)__pyx_v_beta), ((cudnnTensorDescriptor_t)__pyx_v_dstDesc), ((void *)__pyx_v_dstData));
                                                                                                                                                                                                                                                                                                                                             ^
    cupy/cuda/cudnn.cpp: In function ‘PyObject* __pyx_f_4cupy_4cuda_5cudnn_activationBackward_v3(size_t, int, size_t, size_t, size_t, size_t, size_t, size_t, size_t, size_t, size_t, size_t, int)’:
    cupy/cuda/cudnn.cpp:13933:476: error: ‘cudnnActivationBackward_v3’ was not declared in this scope
             __pyx_v_status = cudnnActivationBackward_v3(((cudnnHandle_t)__pyx_v_handle), ((cudnnActivationMode_t)__pyx_v_mode), ((void *)__pyx_v_alpha), ((cudnnTensorDescriptor_t)__pyx_v_srcDesc), ((void *)__pyx_v_srcData), ((cudnnTensorDescriptor_t)__pyx_v_srcDiffDesc), ((void *)__pyx_v_srcDiffData), ((cudnnTensorDescriptor_t)__pyx_v_destDesc), ((void *)__pyx_v_destData), ((void *)__pyx_v_beta), ((cudnnTensorDescriptor_t)__pyx_v_destDiffDesc), ((void *)__pyx_v_destDiffData));
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                ^
    cupy/cuda/cudnn.cpp: In function ‘PyObject* __pyx_f_4cupy_4cuda_5cudnn_activationBackward_v4(size_t, size_t, size_t, size_t, size_t, size_t, size_t, size_t, size_t, size_t, size_t, size_t, int)’:
    cupy/cuda/cudnn.cpp:14194:492: error: ‘cudnnActivationBackward_v4’ was not declared in this scope
             __pyx_v_status = cudnnActivationBackward_v4(((cudnnHandle_t)__pyx_v_handle), ((cudnnActivationDescriptor_t)__pyx_v_activationDesc), ((void *)__pyx_v_alpha), ((cudnnTensorDescriptor_t)__pyx_v_srcDesc), ((void *)__pyx_v_srcData), ((cudnnTensorDescriptor_t)__pyx_v_srcDiffDesc), ((void *)__pyx_v_srcDiffData), ((cudnnTensorDescriptor_t)__pyx_v_destDesc), ((void *)__pyx_v_destData), ((void *)__pyx_v_beta), ((cudnnTensorDescriptor_t)__pyx_v_destDiffDesc), ((void *)__pyx_v_destDiffData));
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                ^
    error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
error
Cleaning up...
  Removing source in /tmp/pip-build-pXxzLQ/chainer
Command "/usr/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-pXxzLQ/chainer/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-U6dnBU-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-build-pXxzLQ/chainer/

What should I do next?
Thank you very much in advance!

@furushchev

This comment has been minimized.

Show comment
Hide comment
@furushchev

furushchev Apr 4, 2017

I checked the version of installed CUDA / cuDNN Version

nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2016 NVIDIA Corporation
Built on Tue_Jan_10_13:22:03_CST_2017
Cuda compilation tools, release 8.0, V8.0.61
// /tmp/cudnn_test.cpp
#include <cudnn.h>
#include <iostream>


int main(int argc, char** argv) {
  std::cout << "CUDNN_VERSION: " << CUDNN_VERSION << std::endl;
  return 0;
}
nvcc /tmp/cudnn_test.cpp -o cudnn_test
./cudnn_test
CUDNN_VERSION: 6020

furushchev commented Apr 4, 2017

I checked the version of installed CUDA / cuDNN Version

nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2016 NVIDIA Corporation
Built on Tue_Jan_10_13:22:03_CST_2017
Cuda compilation tools, release 8.0, V8.0.61
// /tmp/cudnn_test.cpp
#include <cudnn.h>
#include <iostream>


int main(int argc, char** argv) {
  std::cout << "CUDNN_VERSION: " << CUDNN_VERSION << std::endl;
  return 0;
}
nvcc /tmp/cudnn_test.cpp -o cudnn_test
./cudnn_test
CUDNN_VERSION: 6020
@furushchev

This comment has been minimized.

Show comment
Hide comment
@furushchev

furushchev Apr 4, 2017

And here is full log file of sudo pip install -U chainer --no-cache-dir -vvvv

https://gist.github.com/furushchev/829a727419725daa027ecf9174fb6dde

furushchev commented Apr 4, 2017

And here is full log file of sudo pip install -U chainer --no-cache-dir -vvvv

https://gist.github.com/furushchev/829a727419725daa027ecf9174fb6dde

@unnonouno

This comment has been minimized.

Show comment
Hide comment
@unnonouno

unnonouno Apr 4, 2017

Member

cudnn v6 is supported in the master branch and will be availabel in the next version v1.23.0. See #2478.
If you want to use v1.22.0 that is the latest stable version in pipy, use cudnn v5.1.

Member

unnonouno commented Apr 4, 2017

cudnn v6 is supported in the master branch and will be availabel in the next version v1.23.0. See #2478.
If you want to use v1.22.0 that is the latest stable version in pipy, use cudnn v5.1.

@unnonouno unnonouno closed this Apr 4, 2017

@furushchev

This comment has been minimized.

Show comment
Hide comment
@furushchev

furushchev Apr 4, 2017

@unnonouno Thank you for quick response. I overlooked that PIPY version does not support cuDNN v6 yet.

furushchev commented Apr 4, 2017

@unnonouno Thank you for quick response. I overlooked that PIPY version does not support cuDNN v6 yet.

@unnonouno

This comment has been minimized.

Show comment
Hide comment
@unnonouno

unnonouno Apr 4, 2017

Member

Please wait. We'll release the next version soon.

Member

unnonouno commented Apr 4, 2017

Please wait. We'll release the next version soon.

@delta2323 delta2323 modified the milestone: Closed issues and PRs Apr 10, 2017

@sharpstill

This comment has been minimized.

Show comment
Hide comment
@sharpstill

sharpstill Jul 26, 2017

@unnonouno still can not use cudnn v 6.0

import cupy.cudnn
Traceback (most recent call last):
File "", line 1, in
File "/usr/local/anaconda3/lib/python3.6/site-packages/cupy-2.0.0a1-py3.6-linux-ppc64le.egg/cupy/cudnn.py", line 10, in
from cupy.cuda import cudnn
ImportError: /usr/local/anaconda3/lib/python3.6/site-packages/cupy-2.0.0a1-py3.6-linux-ppc64le.egg/cupy/cuda/cudnn.cpython-36m-powerpc64le-linux-gnu.so: undefined symbol: cudnnGetFilterNdDescriptor_v4

sharpstill commented Jul 26, 2017

@unnonouno still can not use cudnn v 6.0

import cupy.cudnn
Traceback (most recent call last):
File "", line 1, in
File "/usr/local/anaconda3/lib/python3.6/site-packages/cupy-2.0.0a1-py3.6-linux-ppc64le.egg/cupy/cudnn.py", line 10, in
from cupy.cuda import cudnn
ImportError: /usr/local/anaconda3/lib/python3.6/site-packages/cupy-2.0.0a1-py3.6-linux-ppc64le.egg/cupy/cuda/cudnn.cpython-36m-powerpc64le-linux-gnu.so: undefined symbol: cudnnGetFilterNdDescriptor_v4

@unnonouno

This comment has been minimized.

Show comment
Hide comment
@unnonouno

unnonouno Jul 26, 2017

Member

Did you reinstall CuPy? cuDNN does not have binary compatibility. When you install newer version of cuDNN, you always need to rebuild CuPy. This error suggests link error with different version of cuDNN.

$ pip uninstall cupy  # Sometimes you have different versions of cupy. Try to uninstall unless it fails
$ pip install cupy --no-cache-dir
Member

unnonouno commented Jul 26, 2017

Did you reinstall CuPy? cuDNN does not have binary compatibility. When you install newer version of cuDNN, you always need to rebuild CuPy. This error suggests link error with different version of cuDNN.

$ pip uninstall cupy  # Sometimes you have different versions of cupy. Try to uninstall unless it fails
$ pip install cupy --no-cache-dir
@JamieShelley

This comment has been minimized.

Show comment
Hide comment
@JamieShelley

JamieShelley Aug 27, 2017

Running easy_install instead of pip did the trick for me - CUDA 7.5 Env hook

JamieShelley commented Aug 27, 2017

Running easy_install instead of pip did the trick for me - CUDA 7.5 Env hook

@sharpstill

This comment has been minimized.

Show comment
Hide comment
@sharpstill

sharpstill Nov 11, 2017

[root@hadoop1 cuda]# ldd cudnn.cpython-36m-powerpc64le-linux-gnu.so
linux-vdso64.so.1 => (0x00003fffa9f60000)
libcudnn.so.5 => not found
libpython3.6m.so.1.0 => /usr/local/anaconda3/lib/libpython3.6m.so.1.0 (0x00003fffa9af0000)
libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x00003fffa9960000)
libm.so.6 => /usr/lib64/libm.so.6 (0x00003fffa9880000)
libgcc_s.so.1 => /usr/lib64/libgcc_s.so.1 (0x00003fffa9840000)
libpthread.so.0 => /usr/lib64/libpthread.so.0 (0x00003fffa9800000)
libc.so.6 => /usr/lib64/libc.so.6 (0x00003fffa9620000)
libdl.so.2 => /usr/lib64/libdl.so.2 (0x00003fffa95f0000)
libutil.so.1 => /usr/lib64/libutil.so.1 (0x00003fffa95c0000)
/lib64/ld64.so.2 (0x000000005fac0000)

sharpstill commented Nov 11, 2017

[root@hadoop1 cuda]# ldd cudnn.cpython-36m-powerpc64le-linux-gnu.so
linux-vdso64.so.1 => (0x00003fffa9f60000)
libcudnn.so.5 => not found
libpython3.6m.so.1.0 => /usr/local/anaconda3/lib/libpython3.6m.so.1.0 (0x00003fffa9af0000)
libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x00003fffa9960000)
libm.so.6 => /usr/lib64/libm.so.6 (0x00003fffa9880000)
libgcc_s.so.1 => /usr/lib64/libgcc_s.so.1 (0x00003fffa9840000)
libpthread.so.0 => /usr/lib64/libpthread.so.0 (0x00003fffa9800000)
libc.so.6 => /usr/lib64/libc.so.6 (0x00003fffa9620000)
libdl.so.2 => /usr/lib64/libdl.so.2 (0x00003fffa95f0000)
libutil.so.1 => /usr/lib64/libutil.so.1 (0x00003fffa95c0000)
/lib64/ld64.so.2 (0x000000005fac0000)

@sharpstill

This comment has been minimized.

Show comment
Hide comment
@sharpstill

sharpstill Nov 11, 2017

Can not found libcudnn.so.5 I have set my LD_LIBRARY_PATH and ldconfig of libcudnn.so path

sharpstill commented Nov 11, 2017

Can not found libcudnn.so.5 I have set my LD_LIBRARY_PATH and ldconfig of libcudnn.so path

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment