Bulding dependens

[BaTpyIIIka]

I use clang 3.8. Can't build.

[ 14%] Building NVCC (Device) object third_party/gloo/gloo/CMakeFiles/gloo_cuda.dir/nccl/gloo_cuda_generated_nccl.cu.o
/usr/local/cuda/include/cuda_fp16.h(3068): error: more than one instance of overloaded function "isinf" matches the argument list:
function "std::isinf(float)"
function "isinf(float)"
argument types are: (float)

/usr/local/cuda/include/cuda_fp16.h(3079): error: more than one instance of overloaded function "isinf" matches the argument list:
function "std::isinf(float)"
function "isinf(float)"
argument types are: (float)

2 errors detected in the compilation of "/tmp/tmpxft_000076a5_00000000-19_nccl.compute_61.cpp1.ii".
CMake Error at gloo_cuda_generated_nccl.cu.o.cmake:262 (message):
Error generating file
/home/loki/caffe2/build/third_party/gloo/gloo/CMakeFiles/gloo_cuda.dir/nccl/./gloo_cuda_generated_nccl.cu.o

[Yangqing]

cc @pietern for gloo.

[mnmoustafa]

I believe I have a similar error on Ubuntu 14.04 on the same 'gloo' but different error. Here is the output:

[ 14%] Building NVCC (Device) object third_party/gloo/gloo/CMakeFiles/gloo_cuda.dir/nccl/./gloo_cuda_generated_nccl.cu.o
/srv/software/caffe2/third_party/gloo/gloo/nccl/nccl.cu(36): error: argument of type "const int *" is incompatible with parameter of type "int *"

1 error detected in the compilation of "/tmp/tmpxft_000015a7_00000000-9_nccl.compute_30.cpp1.ii".
CMake Error at gloo_cuda_generated_nccl.cu.o.cmake:260 (message):
  Error generating file
  /srv/software/caffe2/build/third_party/gloo/gloo/CMakeFiles/gloo_cuda.dir/nccl/./gloo_cuda_generated_nccl.cu.o


make[3]: *** [third_party/gloo/gloo/CMakeFiles/gloo_cuda.dir/nccl/./gloo_cuda_generated_nccl.cu.o] Error 1
make[3]: Leaving directory `/srv/software/caffe2/build'
make[2]: *** [third_party/gloo/gloo/CMakeFiles/gloo_cuda.dir/all] Error 2
make[2]: Leaving directory `/srv/software/caffe2/build'
make[1]: *** [all] Error 2
make[1]: Leaving directory `/srv/software/caffe2/build'
make: *** [all] Error 2

[shuruppino]

@mnmoustafa, @pietern, I had pretty the similar issue today.
After making sure that nvcc --version returns proper CUDA version (8.0 in my case) I've managed to solve it by removing caffe2/build subdir and subsequently calling make again.

[MarcelSimon]

I'm having the same issue, nvcc is at version 8.0. The issue only appears, if an external nccl library is used. Without it, everything works fine

[pietern]

@MarcelSimon Do you have the same issue as @mnmoustafa with the const error? If you use external NCCL, which version of NCCL is that?

[Yangqing]

(closing this since it seems to have become stale, but feel free to re-open if the problem persists.)

[zhangsuosheng]

I have solved the problem by updating my libnccl from 1.2.3 to 2.1.4
before:

after: