Skip to content
This repository has been archived by the owner on Feb 7, 2023. It is now read-only.

Unsupported gpu architecture 'compute_60' #1515

Closed
8greg8 opened this issue Nov 23, 2017 · 3 comments
Closed

Unsupported gpu architecture 'compute_60' #1515

8greg8 opened this issue Nov 23, 2017 · 3 comments
Labels
build

Comments

@8greg8
Copy link

8greg8 commented Nov 23, 2017

Dear all,

I have a problem compiling master branch on centos7 with CUDA version 7.5. I'm using cmake-gui v 3.10 for generating build files. For my environment I'm using Anaconda3 with python 2.7. I disabled ROCKSDB MPI MOBILE OPENGL and METAL. My CUDA ARCH NAME is 50.

First of all, if I compile old caffe with Manual CUDA ARCH NAME where I delete all names that are beyond 50 I can compile just fine.

The problem for caffe2 is as follows:
No matter what CUDA ARCH NAME I choose I always get

[  7%] Building CXX object third_party/gloo/gloo/CMakeFiles/gloo.dir/transport/tcp/pair.cc.o
Compiling src/libwrap.cu                      > /home/gregork/src/caffe2/third_party/nccl/build/obj/libwrap.o
nvcc fatal   : Unsupported gpu architecture 'compute_60'
make[3]: *** [/home/gregork/src/caffe2/third_party/nccl/build/obj/libwrap.o] Error 1
make[2]: *** [nccl_external-prefix/src/nccl_external-stamp/nccl_external-build] Error 2
make[1]: *** [CMakeFiles/nccl_external.dir/all] Error 2
[  7%] Linking CXX static library ../../../lib/libbenchmark.a
[  7%] Built target benchmark
[  7%] Linking CXX static library ../../../lib/libgloo.a
[  7%] Built target python_copy_files
[  7%] Built target gloo
[  7%] Linking CXX static library ../../../lib/libgtest.a
[  7%] Built target gtest
make: *** [all] Error 2

If I choose CUDA ARCH NAME Manual and CUDA ARCH BIN 50 and CUDA ARCH PTX 50. I still get the same error.

Bellow is output of my configuration:

/home/gregork/anaconda3/envs/caffe2/bin/protoc: /home/gregork/anaconda3/envs/caffe2/bin/../lib/libstdc++.so.6: version `GLIBCXX_3.4.20' not found (required by /home/gregork/anaconda3/envs/caffe2/bin/../lib/libprotoc.so.14)
/home/gregork/anaconda3/envs/caffe2/bin/protoc: /home/gregork/anaconda3/envs/caffe2/bin/../lib/libstdc++.so.6: version `CXXABI_1.3.8' not found (required by /home/gregork/anaconda3/envs/caffe2/bin/../lib/libprotoc.so.14)
/home/gregork/anaconda3/envs/caffe2/bin/protoc: /home/gregork/anaconda3/envs/caffe2/bin/../lib/libstdc++.so.6: version `CXXABI_1.3.9' not found (required by /home/gregork/anaconda3/envs/caffe2/bin/../lib/libprotoc.so.14)
/home/gregork/anaconda3/envs/caffe2/bin/protoc: /home/gregork/anaconda3/envs/caffe2/bin/../lib/libstdc++.so.6: version `GLIBCXX_3.4.21' not found (required by /home/gregork/anaconda3/envs/caffe2/bin/../lib/libprotoc.so.14)

/home/gregork/anaconda3/envs/caffe2/bin/protoc: /home/gregork/anaconda3/envs/caffe2/bin/../lib/libstdc++.so.6: version `GLIBCXX_3.4.20' not found (required by /home/gregork/anaconda3/envs/caffe2/bin/../lib/./libprotobuf.so.14)
/home/gregork/anaconda3/envs/caffe2/bin/protoc: /home/gregork/anaconda3/envs/caffe2/bin/../lib/libstdc++.so.6: version `CXXABI_1.3.8' not found (required by /home/gregork/anaconda3/envs/caffe2/bin/../lib/./libprotobuf.so.14)
/home/gregork/anaconda3/envs/caffe2/bin/protoc: /home/gregork/anaconda3/envs/caffe2/bin/../lib/libstdc++.so.6: version `CXXABI_1.3.9' not found (required by /home/gregork/anaconda3/envs/caffe2/bin/../lib/./libprotobuf.so.14)
/home/gregork/anaconda3/envs/caffe2/bin/protoc: /home/gregork/anaconda3/envs/caffe2/bin/../lib/libstdc++.so.6: version `GLIBCXX_3.4.21' not found (required by /home/gregork/anaconda3/envs/caffe2/bin/../lib/./libprotobuf.so.14)

CMake Warning at /home/gregork/local/share/cmake-3.10/Modules/FindProtobuf.cmake:455 (message):
  Protobuf compiler version doesn't match library version 3.4.0
Call Stack (most recent call first):
  cmake/ProtoBuf.cmake:60 (find_package)
  cmake/Dependencies.cmake:8 (include)
  CMakeLists.txt:72 (include)


Using protobuf compiler /home/gregork/anaconda3/envs/caffe2/bin/protoc
The BLAS backend of choice:Eigen
Could NOT find NNPACK (missing: NNPACK_INCLUDE_DIR NNPACK_LIBRARY PTHREADPOOL_LIBRARY) 
Brace yourself, we are building NNPACK
Found PythonInterp: /home/gregork/anaconda3/envs/caffe2/bin/python (found version "2.7.14") 
-- Configuring done

-- Generating done

-- Build files have been written to: /home/gregork/src/caffe2/build/six-download

[100%] Built target six

-- Configuring done

-- Generating done

-- Build files have been written to: /home/gregork/src/caffe2/build/enum-download

[100%] Built target enum

-- Configuring done

-- Generating done

-- Build files have been written to: /home/gregork/src/caffe2/build/opcodes-download

[100%] Built target opcodes

-- Configuring done

-- Generating done

-- Build files have been written to: /home/gregork/src/caffe2/build/peachpy-download

[ 11%] Performing update step for 'peachpy'

Current branch master is up to date.

[ 22%] No configure step for 'peachpy'

[ 33%] No build step for 'peachpy'

[ 44%] No install step for 'peachpy'

[ 55%] No test step for 'peachpy'

[ 66%] Completed 'peachpy'

[100%] Built target peachpy

-- Configuring done

-- Generating done

-- Build files have been written to: /home/gregork/src/caffe2/build/fp16-download

[ 11%] Performing update step for 'fp16'

Current branch master is up to date.

[ 22%] No configure step for 'fp16'

[ 33%] No build step for 'fp16'

[ 44%] No install step for 'fp16'

[ 55%] No test step for 'fp16'

[ 66%] Completed 'fp16'

[100%] Built target fp16

-- Configuring done

-- Generating done

-- Build files have been written to: /home/gregork/src/caffe2/build/fxdiv-download

[ 11%] Performing update step for 'fxdiv'

Current branch master is up to date.

[ 22%] No configure step for 'fxdiv'

[ 33%] No build step for 'fxdiv'

[ 44%] No install step for 'fxdiv'

[ 55%] No test step for 'fxdiv'

[ 66%] Completed 'fxdiv'

[100%] Built target fxdiv

-- Configuring done

-- Generating done

-- Build files have been written to: /home/gregork/src/caffe2/build/psimd-download

[ 11%] Performing update step for 'psimd'

Current branch master is up to date.

[ 22%] No configure step for 'psimd'

[ 33%] No build step for 'psimd'

[ 44%] No install step for 'psimd'

[ 55%] No test step for 'psimd'

[ 66%] Completed 'psimd'

[100%] Built target psimd

-- Configuring done

-- Generating done

-- Build files have been written to: /home/gregork/src/caffe2/build/pthreadpool-download

[ 11%] Performing update step for 'pthreadpool'

Current branch master is up to date.

[ 22%] No configure step for 'pthreadpool'

[ 33%] No build step for 'pthreadpool'

[ 44%] No install step for 'pthreadpool'

[ 55%] No test step for 'pthreadpool'

[ 66%] Completed 'pthreadpool'

[100%] Built target pthreadpool

-- Configuring done

-- Generating done

-- Build files have been written to: /home/gregork/src/caffe2/build/googletest-download

[100%] Built target googletest

-- Configuring done

-- Generating done

-- Build files have been written to: /home/gregork/src/caffe2/build/fxdiv-download

[ 11%] Performing update step for 'fxdiv'

Current branch master is up to date.

[ 22%] No configure step for 'fxdiv'

[ 33%] No build step for 'fxdiv'

[ 44%] No install step for 'fxdiv'

[ 55%] No test step for 'fxdiv'

[ 66%] Completed 'fxdiv'

[100%] Built target fxdiv

-- Configuring done

-- Generating done

-- Build files have been written to: /home/gregork/src/caffe2/build/googletest-download

[100%] Built target googletest

-- Configuring done

-- Generating done

-- Build files have been written to: /home/gregork/src/caffe2/build/googlebenchmark-download

[100%] Built target googlebenchmark

-- Configuring done

-- Generating done

-- Build files have been written to: /home/gregork/src/caffe2/build/googletest-download

[100%] Built target googletest

-- Configuring done

-- Generating done

-- Build files have been written to: /home/gregork/src/caffe2/build/googlebenchmark-download

[100%] Built target googlebenchmark

-- Configuring done

-- Generating done

-- Build files have been written to: /home/gregork/src/caffe2/build/googletest-download

[100%] Built target googletest

-- Configuring done

-- Generating done

-- Build files have been written to: /home/gregork/src/caffe2/build/googlebenchmark-download

[100%] Built target googlebenchmark

Found gflags  (include: /home/gregork/anaconda3/envs/caffe2/include, library: /usr/lib64/libgflags.so)
Found glog    (include: /home/gregork/anaconda3/envs/caffe2/include, library: /usr/lib64/libglog.so)
-- git Version: v0.0.0
-- Version: 0.0.0
-- Performing Test HAVE_STD_REGEX
-- Performing Test HAVE_STD_REGEX
-- Performing Test HAVE_STD_REGEX -- compiled but failed to run
-- Performing Test HAVE_GNU_POSIX_REGEX
-- Performing Test HAVE_GNU_POSIX_REGEX
-- Performing Test HAVE_GNU_POSIX_REGEX -- failed to compile
-- Performing Test HAVE_POSIX_REGEX
-- Performing Test HAVE_POSIX_REGEX
-- Performing Test HAVE_POSIX_REGEX -- success
-- Performing Test HAVE_STEADY_CLOCK
-- Performing Test HAVE_STEADY_CLOCK
-- Performing Test HAVE_STEADY_CLOCK -- success
Found lmdb    (include: /home/gregork/anaconda3/envs/caffe2/include, library: /usr/lib64/liblmdb.so)
Found LevelDB (include: /home/gregork/anaconda3/envs/caffe2/include, library: /usr/lib64/libleveldb.so)
Found Snappy  (include: /home/gregork/anaconda3/envs/caffe2/include, library: /usr/lib64/libsnappy.so)
OpenCV found (/home/gregork/anaconda3/envs/caffe2/share/OpenCV)
CMake Warning at cmake/Dependencies.cmake:241 (find_package):
  By not providing "FindEigen3.cmake" in CMAKE_MODULE_PATH this project has
  asked CMake to find a package configuration file provided by "Eigen3", but
  CMake did not find one.

  Could not find a package configuration file provided by "Eigen3" with any
  of the following names:

    Eigen3Config.cmake
    eigen3-config.cmake

  Add the installation prefix of "Eigen3" to CMAKE_PREFIX_PATH or set
  "Eigen3_DIR" to a directory containing one of the above files.  If "Eigen3"
  provides a separate development package or SDK, be sure it has been
  installed.
Call Stack (most recent call first):
  CMakeLists.txt:72 (include)


Did not find system Eigen. Using third party subdirectory.
Found PythonInterp: /home/gregork/anaconda3/envs/caffe2/bin/python (found suitable version "2.7.14", minimum required is "2.7") 
NumPy ver. 1.12.1 found (include: /home/gregork/anaconda3/envs/caffe2/lib/python2.7/site-packages/numpy/core/include)
Could NOT find pybind11 (missing: pybind11_INCLUDE_DIR) 
CUDA detected: 7.5
Added CUDA NVCC flags for: sm_50 compute_50
Found libcuda: /usr/lib64/libcuda.so
Found libnvrtc: /usr/lib64/libnvrtc.so
Could NOT find CUDNN (missing: CUDNN_LIBRARY) 
Could NOT find NCCL (missing: NCCL_INCLUDE_DIRS NCCL_LIBRARIES) 
NCCL: /home/gregork/src/caffe2/third_party/nccl/build/lib/libnccl_static.a
Could NOT find CUB (missing: CUB_INCLUDE_DIR) 
Could NOT find Gloo (missing: Gloo_INCLUDE_DIR Gloo_LIBRARY) 
CUDA detected: 7.5
Found libcuda: /usr/lib64/libcuda.so
Found libnvrtc: /usr/lib64/libnvrtc.so
Does not need to define long separately.
Current compiler supports avx2 extention. Will build perfkernels.
GCC 4.8.5: Adding gcc and gcc_s libs to link line
Include NCCL operators
Include NNPACK operators
Including image processing operators
Excluding video processing operators due to no opencv
Excluding mkl operators as we are not using mkl
MPI operators skipped due to no MPI support
Include Observer library
Automatically generating missing __init__.py files.

******** Summary ********
General:
  Git version           : v0.8.1-715-ged1b7d9b
  System                : Linux
  C++ compiler          : /home/gregork/anaconda3/envs/caffe2/bin/c++
  C++ compiler version  : 4.8.5
  Protobuf compiler     : /home/gregork/anaconda3/envs/caffe2/bin/protoc
  CXX flags             :  -std=c++11 -O2 -fPIC -Wno-narrowing
  Build type            : Release
  Compile definitions   : 

  BUILD_BINARY          : ON
  BUILD_PYTHON          : ON
    Python version      : 2.7.14
    Python library      : /home/gregork/anaconda3/envs/caffe2/lib/libpython2.7.so
  BUILD_SHARED_LIBS     : ON
  BUILD_TEST            : ON
  USE_ATEN              : OFF
  USE_ASAN              : OFF
  USE_CUDA              : ON
    CUDA version        : 7.5
    CuDNN version       : 
  USE_EIGEN_FOR_BLAS    : 1
  USE_FFMPEG            : OFF
  USE_GFLAGS            : ON
  USE_GLOG              : ON
  USE_GLOO              : ON
  USE_LEVELDB           : ON
    LevelDB version     : 1.20
    Snappy version      : ..
  USE_LITE_PROTO        : OFF
  USE_LMDB              : ON
    LMDB version        : 0.9.21
  USE_METAL             : OFF
  USE_MKL               : 
  USE_MOBILE_OPENGL     : OFF
  USE_MPI               : OFF
  USE_NCCL              : ON
  USE_NERVANA_GPU       : OFF
  USE_NNPACK            : ON
  USE_OBSERVERS         : ON
  USE_OPENCV            : ON
    OpenCV version      : 3.1.0
  USE_OPENMP            : OFF
  USE_REDIS             : OFF
  USE_ROCKSDB           : OFF
  USE_THREADS           : ON
  USE_ZMQ               : OFF
Configuring done
@iteal
Copy link

iteal commented Nov 24, 2017 •
edited

You need to update CUDA to version 8, look at this page:
http://docs.nvidia.com/cuda/pascal-compatibility-guide/index.html

@pietern
Copy link
Contributor

pietern commented Dec 12, 2017

This is caused by the external Makefile in third-party/nccl. If you don't care about NCCL you can disable it with -DUSE_NCCL=OFF. Otherwise you will have to modify its Makefile.

Another option is to update CUDA to version 8 or 9 as @laetitiaoist suggests.

@pietern pietern closed this as completed Dec 12, 2017
@pietern pietern added the build label Dec 12, 2017
@Haiyang21
Copy link

You can comment the codes about "compute_60" or larger compute in the file "third_party/nccl/Makefile"

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
build
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@pietern @Haiyang21 @8greg8 @iteal