New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

How to get GPU architecture at build/runtime stage? #597

Closed
peetonn opened this Issue Jan 9, 2018 · 4 comments

Comments

Projects
None yet
2 participants
@peetonn

peetonn commented Jan 9, 2018

Trying to build image with OpenPose, however it fails with error:

nvcc fatal   : Unsupported gpu architecture 'compute_20'
CMake Error at cuda_compile_generated_tanh_layer.cu.o.cmake:207 (message):
  Error generating
  /openpose/build/caffe/src/openpose_caffe-build/src/caffe/CMakeFiles/cuda_compile.dir/layers/./cuda_compile_generated_tanh_layer.cu.o

src/caffe/CMakeFiles/caffe.dir/build.make:476: recipe for target 'src/caffe/CMakeFiles/cuda_compile.dir/layers/cuda_compile_generated_tanh_layer.cu.o' failed
make[5]: *** [src/caffe/CMakeFiles/cuda_compile.dir/layers/cuda_compile_generated_tanh_layer.cu.o] Error 1
CMakeFiles/Makefile2:272: recipe for target 'src/caffe/CMakeFiles/caffe.dir/all' failed
make[4]: *** [src/caffe/CMakeFiles/caffe.dir/all] Error 2
Makefile:127: recipe for target 'all' failed
make[3]: *** [all] Error 2
CMakeFiles/openpose_caffe.dir/build.make:110: recipe for target 'caffe/src/openpose_caffe-stamp/openpose_caffe-build' failed
make[2]: *** [caffe/src/openpose_caffe-stamp/openpose_caffe-build] Error 2
CMakeFiles/Makefile2:67: recipe for target 'CMakeFiles/openpose_caffe.dir/all' failed
make[1]: *** [CMakeFiles/openpose_caffe.dir/all] Error 2
Makefile:127: recipe for target 'all' failed
make: *** [all] Error 2

I think this happened because during configuration step, automatic GPU detection failed:

-- CUDA detected: 9.1
-- Found cuDNN: ver. 5.1.10 found (include: /usr/local/cuda/include, library: /usr/local/cuda/lib64/libcudnn.so)
-- Automatic GPU detection failed. Building for all known architectures.
-- Added CUDA NVCC flags for: sm_20 sm_21 sm_30 sm_35 sm_50 sm_60 sm_61

And this happens both when I do docker build with the Dockerfile provided below or when I run commands step-by-step interactively inside the container.

Dockerfile:

FROM peetonn/ice-ar-base:latest
LABEL maintainer "Peter Gusev <peter@remap.ucla.edu>"
ARG DEBUG_VER=unknown

RUN git clone --recursive https://github.com/CMU-Perceptual-Computing-Lab/openpose.git \
	&& cd /openpose \
	&& yes | bash ./ubuntu/install_cmake.sh \
	&& mkdir -p /openpose/build \
	&& cd /openpose/build \
	&& cmake .. \
	&& make -j$(nproc) \
	&& make install

RUN DEBUG_VER=${DEBUG_VER} git clone --recursive https://github.com/remap/ice-ar
RUN cd /ice-ar/edge/openpose \
	&& g++ -std=c++11 ice-openpose.cpp ipc-shim.c pipe-utils.c jsonOsstream.cpp \
	  -o ice-openpose \
	  -L/usr/local/lib  -lopenpose -lpthread -lopencv_highgui \
	  -lopencv_objdetect -lopencv_imgcodecs -lopencv_imgproc \
	  -lopencv_core -lgflags -lnanomsg

RUN apt-get remove -y git cmake wget lsb-release python-pip
@flx42

This comment has been minimized.

Member

flx42 commented Jan 9, 2018

Since you are using Caffe, you can do something like this:
https://github.com/NVIDIA/gpu-rest-engine/blob/master/Dockerfile.caffe_server#L47

At build time, you don't get GPU access by default, that's why the detection fails.

@flx42 flx42 added the question label Jan 9, 2018

@flx42 flx42 closed this Jan 9, 2018

@peetonn

This comment has been minimized.

peetonn commented Jan 9, 2018

How shall I setup these variables CUDA_ARCH_BIN and CUDA_ARCH_PTX ?
Similarly to the Dockerfile you provided?

ENV CUDA_ARCH_BIN "35 52 60 61 70"
ENV CUDA_ARCH_PTX "70"
@peetonn

This comment has been minimized.

peetonn commented Jan 9, 2018

Cmake complains with:

CMake Warning:
  Manually-specified variables were not used by the project:

    CUDA_ARCH_BIN
    CUDA_ARCH_NAME

when I run cmake -DCUDA_ARCH_NAME=Manual -DCUDA_ARCH_BIN="${CUDA_ARCH_BIN}" -DCUDA_ARCH_PTX="${CUDA_ARCH_PTX}" ..

@flx42

This comment has been minimized.

Member

flx42 commented Jan 9, 2018

These settings needs to be passed to the Caffe build, not to the top-level cmake command of your project.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment