failed to install chainer with cuda 8.0 / cuDNN v6

[furushchev]

Hi,

I installed CUDA Toolkit 8.0 and cuDNN v6.
I then tried to install chainer, but it failed with errors as below:

    cupy/cuda/cudnn.cpp: In function ‘PyObject* __pyx_f_4cupy_4cuda_5cudnn_setFilterNdDescriptor_v3(size_t, int, int, size_t, int)’:
    cupy/cuda/cudnn.cpp:4203:177: error: ‘cudnnSetFilterNdDescriptor_v3’ was not declared in this scope
       __pyx_v_status = cudnnSetFilterNdDescriptor_v3(((cudnnFilterDescriptor_t)__pyx_v_filterDesc), ((cudnnDataType_t)__pyx_v_dataType), __pyx_v_nbDims, ((int *)__pyx_v_filterDimA));
                                                                                                                                                                                     ^
    cupy/cuda/cudnn.cpp: In function ‘PyObject* __pyx_f_4cupy_4cuda_5cudnn_setFilter4dDescriptor_v4(size_t, int, int, int, int, int, int, int)’:
    cupy/cuda/cudnn.cpp:4359:215: error: ‘cudnnSetFilter4dDescriptor_v4’ was not declared in this scope
       __pyx_v_status = cudnnSetFilter4dDescriptor_v4(((cudnnFilterDescriptor_t)__pyx_v_filterDesc), ((cudnnDataType_t)__pyx_v_dataType), ((cudnnTensorFormat_t)__pyx_v_format), __pyx_v_k, __pyx_v_c, __pyx_v_h, __pyx_v_w);
                                                                                                                                                                                                                           ^
    cupy/cuda/cudnn.cpp: In function ‘PyObject* __pyx_f_4cupy_4cuda_5cudnn_setFilterNdDescriptor_v4(size_t, int, int, int, size_t, int)’:
    cupy/cuda/cudnn.cpp:4542:216: error: ‘cudnnSetFilterNdDescriptor_v4’ was not declared in this scope
       __pyx_v_status = cudnnSetFilterNdDescriptor_v4(((cudnnFilterDescriptor_t)__pyx_v_filterDesc), ((cudnnDataType_t)__pyx_v_dataType), ((cudnnTensorFormat_t)__pyx_v_format), __pyx_v_nbDims, ((int *)__pyx_v_filterDimA));
                                                                                                                                                                                                                            ^
    cupy/cuda/cudnn.cpp: In function ‘PyObject* __pyx_f_4cupy_4cuda_5cudnn_getFilterNdDescriptor(size_t, int, int)’:
    cupy/cuda/cudnn.cpp:4729:202: error: ‘cudnnGetFilterNdDescriptor_v4’ was not declared in this scope
       __pyx_v_status = cudnnGetFilterNdDescriptor_v4(((cudnnFilterDescriptor_t)__pyx_v_wDesc), __pyx_v_nbDimsRequested, (&__pyx_v_dataType), (&__pyx_v_format), (&__pyx_v_nbDims), (&(__pyx_v_filterDimA[0])));
                                                                                                                                                                                                              ^
    cupy/cuda/cudnn.cpp: In function ‘PyObject* __pyx_f_4cupy_4cuda_5cudnn_setConvolution2dDescriptor_v4(size_t, int, int, int, int, int, int, int, int)’:
    cupy/cuda/cudnn.cpp:5112:231: error: too few arguments to function ‘cudnnStatus_t cudnnSetConvolution2dDescriptor(cudnnConvolutionDescriptor_t, int, int, int, int, int, int, cudnnConvolutionMode_t, cudnnDataType_t)’
       __pyx_v_status = cudnnSetConvolution2dDescriptor_v4(((cudnnConvolutionDescriptor_t)__pyx_v_convDesc), __pyx_v_pad_h, __pyx_v_pad_w, __pyx_v_u, __pyx_v_v, __pyx_v_upscalex, __pyx_v_upscaley, ((cudnnConvolutionMode_t)__pyx_v_mode));
                                                                                                                                                                                                                                           ^
    In file included from cupy/cuda/cupy_cudnn.h:10:0,
                     from cupy/cuda/cudnn.cpp:481:
    /usr/local/cuda/include/cudnn.h:500:27: note: declared here
     cudnnStatus_t CUDNNWINAPI cudnnSetConvolution2dDescriptor( cudnnConvolutionDescriptor_t convDesc,
                               ^
    cupy/cuda/cudnn.cpp: In function ‘PyObject* __pyx_f_4cupy_4cuda_5cudnn_setPooling2dDescriptor_v3(size_t, int, int, int, int, int, int, int, int)’:
    cupy/cuda/cudnn.cpp:10226:279: error: ‘cudnnSetPooling2dDescriptor_v3’ was not declared in this scope
       __pyx_v_status = cudnnSetPooling2dDescriptor_v3(((cudnnPoolingDescriptor_t)__pyx_v_poolingDesc), ((cudnnPoolingMode_t)__pyx_v_mode), __pyx_v_windowHeight, __pyx_v_windowWidth, __pyx_v_verticalPadding, __pyx_v_horizontalPadding, __pyx_v_verticalStride, __pyx_v_horizontalStride);
                                                                                                                                                                                                                                                                                           ^
    cupy/cuda/cudnn.cpp: In function ‘PyObject* __pyx_f_4cupy_4cuda_5cudnn_setPooling2dDescriptor_v4(size_t, int, int, int, int, int, int, int, int, int)’:
    cupy/cuda/cudnn.cpp:10418:330: error: ‘cudnnSetPooling2dDescriptor_v4’ was not declared in this scope
       __pyx_v_status = cudnnSetPooling2dDescriptor_v4(((cudnnPoolingDescriptor_t)__pyx_v_poolingDesc), ((cudnnPoolingMode_t)__pyx_v_mode), ((cudnnNanPropagation_t)__pyx_v_maxpoolingNanOpt), __pyx_v_windowHeight, __pyx_v_windowWidth, __pyx_v_verticalPadding, __pyx_v_horizontalPadding, __pyx_v_verticalStride, __pyx_v_horizontalStride);
                                                                                                                                                                                                                                                                                                                                              ^
    cupy/cuda/cudnn.cpp: In function ‘PyObject* __pyx_f_4cupy_4cuda_5cudnn_setPoolingNdDescriptor_v3(size_t, int, int, size_t, size_t, size_t, int)’:
    cupy/cuda/cudnn.cpp:10619:232: error: ‘cudnnSetPoolingNdDescriptor_v3’ was not declared in this scope
       __pyx_v_status = cudnnSetPoolingNdDescriptor_v3(((cudnnPoolingDescriptor_t)__pyx_v_poolingDesc), ((cudnnPoolingMode_t)__pyx_v_mode), __pyx_v_nbDims, ((int *)__pyx_v_windowDimA), ((int *)__pyx_v_paddingA), ((int *)__pyx_v_strideA));
                                                                                                                                                                                                                                            ^
    cupy/cuda/cudnn.cpp: In function ‘PyObject* __pyx_f_4cupy_4cuda_5cudnn_setPoolingNdDescriptor_v4(size_t, int, int, int, size_t, size_t, size_t, int)’:
    cupy/cuda/cudnn.cpp:10793:283: error: ‘cudnnSetPoolingNdDescriptor_v4’ was not declared in this scope
       __pyx_v_status = cudnnSetPoolingNdDescriptor_v4(((cudnnPoolingDescriptor_t)__pyx_v_poolingDesc), ((cudnnPoolingMode_t)__pyx_v_mode), ((cudnnNanPropagation_t)__pyx_v_maxpoolingNanOpt), __pyx_v_nbDims, ((int *)__pyx_v_windowDimA), ((int *)__pyx_v_paddingA), ((int *)__pyx_v_strideA));
                                                                                                                                                                                                                                                                                               ^
    cupy/cuda/cudnn.cpp: In function ‘PyObject* __pyx_f_4cupy_4cuda_5cudnn_activationForward_v3(size_t, int, size_t, size_t, size_t, size_t, size_t, size_t, int)’:
    cupy/cuda/cudnn.cpp:13483:313: error: ‘cudnnActivationForward_v3’ was not declared in this scope
             __pyx_v_status = cudnnActivationForward_v3(((cudnnHandle_t)__pyx_v_handle), ((cudnnActivationMode_t)__pyx_v_mode), ((void *)__pyx_v_alpha), ((cudnnTensorDescriptor_t)__pyx_v_srcDesc), ((void *)__pyx_v_srcData), ((void *)__pyx_v_beta), ((cudnnTensorDescriptor_t)__pyx_v_dstDesc), ((void *)__pyx_v_dstData));
                                                                                                                                                                                                                                                                                                                             ^
    cupy/cuda/cudnn.cpp: In function ‘PyObject* __pyx_f_4cupy_4cuda_5cudnn_activationForward_v4(size_t, size_t, size_t, size_t, size_t, size_t, size_t, size_t, int)’:
    cupy/cuda/cudnn.cpp:13708:329: error: ‘cudnnActivationForward_v4’ was not declared in this scope
             __pyx_v_status = cudnnActivationForward_v4(((cudnnHandle_t)__pyx_v_handle), ((cudnnActivationDescriptor_t)__pyx_v_activationDesc), ((void *)__pyx_v_alpha), ((cudnnTensorDescriptor_t)__pyx_v_srcDesc), ((void *)__pyx_v_srcData), ((void *)__pyx_v_beta), ((cudnnTensorDescriptor_t)__pyx_v_dstDesc), ((void *)__pyx_v_dstData));
                                                                                                                                                                                                                                                                                                                                             ^
    cupy/cuda/cudnn.cpp: In function ‘PyObject* __pyx_f_4cupy_4cuda_5cudnn_activationBackward_v3(size_t, int, size_t, size_t, size_t, size_t, size_t, size_t, size_t, size_t, size_t, size_t, int)’:
    cupy/cuda/cudnn.cpp:13933:476: error: ‘cudnnActivationBackward_v3’ was not declared in this scope
             __pyx_v_status = cudnnActivationBackward_v3(((cudnnHandle_t)__pyx_v_handle), ((cudnnActivationMode_t)__pyx_v_mode), ((void *)__pyx_v_alpha), ((cudnnTensorDescriptor_t)__pyx_v_srcDesc), ((void *)__pyx_v_srcData), ((cudnnTensorDescriptor_t)__pyx_v_srcDiffDesc), ((void *)__pyx_v_srcDiffData), ((cudnnTensorDescriptor_t)__pyx_v_destDesc), ((void *)__pyx_v_destData), ((void *)__pyx_v_beta), ((cudnnTensorDescriptor_t)__pyx_v_destDiffDesc), ((void *)__pyx_v_destDiffData));
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                ^
    cupy/cuda/cudnn.cpp: In function ‘PyObject* __pyx_f_4cupy_4cuda_5cudnn_activationBackward_v4(size_t, size_t, size_t, size_t, size_t, size_t, size_t, size_t, size_t, size_t, size_t, size_t, int)’:
    cupy/cuda/cudnn.cpp:14194:492: error: ‘cudnnActivationBackward_v4’ was not declared in this scope
             __pyx_v_status = cudnnActivationBackward_v4(((cudnnHandle_t)__pyx_v_handle), ((cudnnActivationDescriptor_t)__pyx_v_activationDesc), ((void *)__pyx_v_alpha), ((cudnnTensorDescriptor_t)__pyx_v_srcDesc), ((void *)__pyx_v_srcData), ((cudnnTensorDescriptor_t)__pyx_v_srcDiffDesc), ((void *)__pyx_v_srcDiffData), ((cudnnTensorDescriptor_t)__pyx_v_destDesc), ((void *)__pyx_v_destData), ((void *)__pyx_v_beta), ((cudnnTensorDescriptor_t)__pyx_v_destDiffDesc), ((void *)__pyx_v_destDiffData));
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                ^
    error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
error
Cleaning up...
  Removing source in /tmp/pip-build-pXxzLQ/chainer
Command "/usr/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-pXxzLQ/chainer/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-U6dnBU-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-build-pXxzLQ/chainer/

What should I do next?
Thank you very much in advance!

[furushchev]

I checked the version of installed CUDA / cuDNN Version

nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2016 NVIDIA Corporation
Built on Tue_Jan_10_13:22:03_CST_2017
Cuda compilation tools, release 8.0, V8.0.61

// /tmp/cudnn_test.cpp
#include <cudnn.h>
#include <iostream>


int main(int argc, char** argv) {
  std::cout << "CUDNN_VERSION: " << CUDNN_VERSION << std::endl;
  return 0;
}

nvcc /tmp/cudnn_test.cpp -o cudnn_test
./cudnn_test
CUDNN_VERSION: 6020

[furushchev]

And here is full log file of sudo pip install -U chainer --no-cache-dir -vvvv

https://gist.github.com/furushchev/829a727419725daa027ecf9174fb6dde

[unnonouno]

cudnn v6 is supported in the master branch and will be availabel in the next version v1.23.0. See #2478.
If you want to use v1.22.0 that is the latest stable version in pipy, use cudnn v5.1.

[furushchev]

@unnonouno Thank you for quick response. I overlooked that PIPY version does not support cuDNN v6 yet.

[unnonouno]

Please wait. We'll release the next version soon.

[machanic]

@unnonouno still can not use cudnn v 6.0

import cupy.cudnn
Traceback (most recent call last):
File "", line 1, in
File "/usr/local/anaconda3/lib/python3.6/site-packages/cupy-2.0.0a1-py3.6-linux-ppc64le.egg/cupy/cudnn.py", line 10, in
from cupy.cuda import cudnn
ImportError: /usr/local/anaconda3/lib/python3.6/site-packages/cupy-2.0.0a1-py3.6-linux-ppc64le.egg/cupy/cuda/cudnn.cpython-36m-powerpc64le-linux-gnu.so: undefined symbol: cudnnGetFilterNdDescriptor_v4

[unnonouno]

Did you reinstall CuPy? cuDNN does not have binary compatibility. When you install newer version of cuDNN, you always need to rebuild CuPy. This error suggests link error with different version of cuDNN.

$ pip uninstall cupy  # Sometimes you have different versions of cupy. Try to uninstall unless it fails
$ pip install cupy --no-cache-dir

[JamieShelley]

Running easy_install instead of pip did the trick for me - CUDA 7.5 Env hook

[machanic]

[root@hadoop1 cuda]# ldd cudnn.cpython-36m-powerpc64le-linux-gnu.so
linux-vdso64.so.1 => (0x00003fffa9f60000)
libcudnn.so.5 => not found
libpython3.6m.so.1.0 => /usr/local/anaconda3/lib/libpython3.6m.so.1.0 (0x00003fffa9af0000)
libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x00003fffa9960000)
libm.so.6 => /usr/lib64/libm.so.6 (0x00003fffa9880000)
libgcc_s.so.1 => /usr/lib64/libgcc_s.so.1 (0x00003fffa9840000)
libpthread.so.0 => /usr/lib64/libpthread.so.0 (0x00003fffa9800000)
libc.so.6 => /usr/lib64/libc.so.6 (0x00003fffa9620000)
libdl.so.2 => /usr/lib64/libdl.so.2 (0x00003fffa95f0000)
libutil.so.1 => /usr/lib64/libutil.so.1 (0x00003fffa95c0000)
/lib64/ld64.so.2 (0x000000005fac0000)

[machanic]

Can not found libcudnn.so.5 I have set my LD_LIBRARY_PATH and ldconfig of libcudnn.so path