build pytorch from source on ubuntu, building error from fbgemm::SparseAdaGradSignature

[CnybTseng]

Compiling  functions.cu                        > /data/tseng/toolkit/pytorch/build/nccl/obj/collectives/device/functions.o
[ 67%] Building CXX object caffe2/CMakeFiles/torch_cpu.dir/queue/rebatching_queue.cc.o
[ 67%] Building CXX object caffe2/CMakeFiles/torch_cpu.dir/queue/rebatching_queue_ops.cc.o
[ 67%] Building CXX object caffe2/CMakeFiles/torch_cpu.dir/sgd/adadelta_op.cc.o
[ 67%] Building CXX object caffe2/CMakeFiles/torch_cpu.dir/sgd/adagrad_fused.cc.o
[ 67%] Building CXX object caffe2/CMakeFiles/torch_cpu.dir/sgd/adagrad_op.cc.o
[ 67%] Building CXX object caffe2/CMakeFiles/torch_cpu.dir/sgd/adam_op.cc.o
[ 67%] Building CXX object caffe2/CMakeFiles/torch_cpu.dir/sgd/clip_tensor_op.cc.o
[ 67%] Building CXX object caffe2/CMakeFiles/torch_cpu.dir/sgd/ftrl_op.cc.o
[ 67%] Building CXX object caffe2/CMakeFiles/torch_cpu.dir/sgd/gftrl_op.cc.o
[ 67%] Building CXX object caffe2/CMakeFiles/torch_cpu.dir/sgd/iter_op.cc.o
In file included from /data/tseng/toolkit/pytorch/caffe2/sgd/adagrad_fused.h:3:0,
                 from /data/tseng/toolkit/pytorch/caffe2/sgd/adagrad_fused.cc:1:
/data/tseng/toolkit/pytorch/caffe2/sgd/adagrad_op.h: In member function 'bool caffe2::SparseAdagradOp::DoRunWithType()':
/data/tseng/toolkit/pytorch/caffe2/sgd/adagrad_op.h:253:16: error: no match for call to '(fbgemm::SparseAdaGradSignature<int>::Type {aka std::function<int(int, long unsigned int, float*, const float*, float*, const int*, float, float, float)>}) (long int&, int64_t, float*&, const float*&, float*&, const int32_t*, float&, const float& '
           lr[0]);
                ^
In file included from /data/tseng/toolkit/pytorch/caffe2/core/operator.h:9:0,
                 from /data/tseng/toolkit/pytorch/caffe2/sgd/adagrad_op.h:3,
                 from /data/tseng/toolkit/pytorch/caffe2/sgd/adagrad_fused.h:3,
                 from /data/tseng/toolkit/pytorch/caffe2/sgd/adagrad_fused.cc:1:
/usr/include/c++/5/functional:2262:5: note: candidate: _Res std::function<_Res(_ArgTypes ...)>::operator()(_ArgTypes ...) const [with _Res = int; _ArgTypes = {int, long unsigned int, float*, const float*, float*, const int*, float, float, float}]
     function<_Res(_ArgTypes...)>::
     ^
/usr/include/c++/5/functional:2262:5: note:   candidate expects 9 arguments, 8 provided
In file included from /data/tseng/toolkit/pytorch/caffe2/sgd/adagrad_fused.h:3:0,
                 from /data/tseng/toolkit/pytorch/caffe2/sgd/adagrad_fused.cc:1:
/data/tseng/toolkit/pytorch/caffe2/sgd/adagrad_op.h:263:16: error: no match for call to '(fbgemm::SparseAdaGradSignature<long int>::Type {aka std::function<int(int, long unsigned int, float*, const float*, float*, const long int*, float, float, float)>}) (long int&, int64_t, float*&, const float*&, float*&, const int64_t*, float&, const float&)'
           lr[0]);
                ^
In file included from /data/tseng/toolkit/pytorch/caffe2/core/operator.h:9:0,
                 from /data/tseng/toolkit/pytorch/caffe2/sgd/adagrad_op.h:3,
                 from /data/tseng/toolkit/pytorch/caffe2/sgd/adagrad_fused.h:3,
                 from /data/tseng/toolkit/pytorch/caffe2/sgd/adagrad_fused.cc:1:
/usr/include/c++/5/functional:2262:5: note: candidate: _Res std::function<_Res(_ArgTypes ...)>::operator()(_ArgTypes ...) const [with _Res = int; _ArgTypes = {int, long unsigned int, float*, const float*, float*, const long int*, float, float, float}]
     function<_Res(_ArgTypes...)>::
     ^
/usr/include/c++/5/functional:2262:5: note:   candidate expects 9 arguments, 8 provided
In file included from /data/tseng/toolkit/pytorch/caffe2/sgd/adagrad_op.cc:1:0:
/data/tseng/toolkit/pytorch/caffe2/sgd/adagrad_op.h: In member function 'bool caffe2::SparseAdagradOp::DoRunWithType()':
/data/tseng/toolkit/pytorch/caffe2/sgd/adagrad_op.h:253:16: error: no match for call to '(fbgemm::SparseAdaGradSignature<int>::Type {aka std::function<int(int, long unsigned int, float*, const float*, float*, const int*, float, float, float)>}) (long int&, int64_t, float*&, const float*&, float*&, const int32_t*, float&, const float& '
           lr[0]);
                ^
In file included from /data/tseng/toolkit/pytorch/caffe2/core/operator.h:9:0,
                 from /data/tseng/toolkit/pytorch/caffe2/sgd/adagrad_op.h:3,
                 from /data/tseng/toolkit/pytorch/caffe2/sgd/adagrad_op.cc:1:
/usr/include/c++/5/functional:2262:5: note: candidate: _Res std::function<_Res(_ArgTypes ...)>::operator()(_ArgTypes ...) const [with _Res = int; _ArgTypes = {int, long unsigned int, float*, const float*, float*, const int*, float, float, float}]
     function<_Res(_ArgTypes...)>::
     ^
/usr/include/c++/5/functional:2262:5: note:   candidate expects 9 arguments, 8 provided
In file included from /data/tseng/toolkit/pytorch/caffe2/sgd/adagrad_op.cc:1:0:
/data/tseng/toolkit/pytorch/caffe2/sgd/adagrad_op.h:263:16: error: no match for call to '(fbgemm::SparseAdaGradSignature<long int>::Type {aka std::function<int(int, long unsigned int, float*, const float*, float*, const long int*, float, float, float)>}) (long int&, int64_t, float*&, const float*&, float*&, const int64_t*, float&, const float&)'
           lr[0]);
                ^
In file included from /data/tseng/toolkit/pytorch/caffe2/core/operator.h:9:0,
                 from /data/tseng/toolkit/pytorch/caffe2/sgd/adagrad_op.h:3,
                 from /data/tseng/toolkit/pytorch/caffe2/sgd/adagrad_op.cc:1:
/usr/include/c++/5/functional:2262:5: note: candidate: _Res std::function<_Res(_ArgTypes ...)>::operator()(_ArgTypes ...) const [with _Res = int; _ArgTypes = {int, long unsigned int, float*, const float*, float*, const long int*, float, float, float}]
     function<_Res(_ArgTypes...)>::
     ^
/usr/include/c++/5/functional:2262:5: note:   candidate expects 9 arguments, 8 provided
[ 67%] Building CXX object caffe2/CMakeFiles/torch_cpu.dir/sgd/lars_op.cc.o
In file included from /data/tseng/toolkit/pytorch/caffe2/sgd/adagrad_op.cc:1:0:
/data/tseng/toolkit/pytorch/caffe2/sgd/adagrad_op.h: In instantiation of 'bool caffe2::RowWiseSparseAdagradOp<Context>::DoRunWithType() [with SIndex = int; Context = caffe2::CPUContext]':
/data/tseng/toolkit/pytorch/caffe2/core/operator.h:1302:1:   required from 'static bool caffe2::DispatchHelper<caffe2::TensorTypes<FirstType, Types ...>, ExtraArgs ...>::call(Op*, const caffe2::TypeMeta&) [with Op = caffe2::RowWiseSparseAdagradOp<caffe2::CPUContext>; FirstType = int; Types = {long int}; ExtraArgs = {}]'
/data/tseng/toolkit/pytorch/caffe2/core/operator.h:1302:1:   required from 'static bool caffe2::DispatchHelper<caffe2::TensorTypes<FirstType, Types ...>, ExtraArgs ...>::call(Op*, const caffe2::Tensor&) [with Op = caffe2::RowWiseSparseAdagradOp<caffe2::CPUContext>; FirstType = int; Types = {long int}; ExtraArgs = {}]'
/data/tseng/toolkit/pytorch/caffe2/sgd/adagrad_op.h:376:63:   required from 'bool caffe2::RowWiseSparseAdagradOp<Context>::RunOnDevice() [with Context = caffe2::CPUContext]'
/data/tseng/toolkit/pytorch/caffe2/sgd/adagrad_op.cc:215:1:   required from here
/data/tseng/toolkit/pytorch/caffe2/sgd/adagrad_op.h:434:26: error: no match for call to '(fbgemm::SparseAdaGradSignature<int>::Type {aka std::function<int(int, long unsigned int, float*, const float*, float*, const int*, float, float, float)>}) (long int&, int64_t, float*&, const float*&, float*&, const int*, float&, const float&)'
       num_rows_processed = kernel_i32_(
                          ^
In file included from /data/tseng/toolkit/pytorch/caffe2/core/operator.h:9:0,
                 from /data/tseng/toolkit/pytorch/caffe2/sgd/adagrad_op.h:3,
                 from /data/tseng/toolkit/pytorch/caffe2/sgd/adagrad_op.cc:1:
/usr/include/c++/5/functional:2262:5: note: candidate: _Res std::function<_Res(_ArgTypes ...)>::operator()(_ArgTypes ...) const [with _Res = int; _ArgTypes = {int, long unsigned int, float*, const float*, float*, const int*, float, float, float}]
     function<_Res(_ArgTypes...)>::
     ^
/usr/include/c++/5/functional:2262:5: note:   candidate expects 9 arguments, 8 provided
In file included from /data/tseng/toolkit/pytorch/caffe2/sgd/adagrad_op.cc:1:0:
/data/tseng/toolkit/pytorch/caffe2/sgd/adagrad_op.h:444:26: error: no match for call to '(fbgemm::SparseAdaGradSignature<long int>::Type {aka std::function<int(int, long unsigned int, float*, const float*, float*, const long int*, float, float, float)>}) (long int&, int64_t, float*&, const float*&, float*&, const int64_t*, float&, const float&)'
       num_rows_processed = kernel_i64_(
                          ^
In file included from /data/tseng/toolkit/pytorch/caffe2/core/operator.h:9:0,
                 from /data/tseng/toolkit/pytorch/caffe2/sgd/adagrad_op.h:3,
                 from /data/tseng/toolkit/pytorch/caffe2/sgd/adagrad_op.cc:1:
/usr/include/c++/5/functional:2262:5: note: candidate: _Res std::function<_Res(_ArgTypes ...)>::operator()(_ArgTypes ...) const [with _Res = int; _ArgTypes = {int, long unsigned int, float*, const float*, float*, const long int*, float, float, float}]
     function<_Res(_ArgTypes...)>::
     ^
/usr/include/c++/5/functional:2262:5: note:   candidate expects 9 arguments, 8 provided
In file included from /data/tseng/toolkit/pytorch/caffe2/sgd/adagrad_op.cc:1:0:
/data/tseng/toolkit/pytorch/caffe2/sgd/adagrad_op.h: In instantiation of 'bool caffe2::RowWiseSparseAdagradOp<Context>::DoRunWithType() [with SIndex = long int; Context = caffe2::CPUContext]':
/data/tseng/toolkit/pytorch/caffe2/core/operator.h:1302:1:   required from 'static bool caffe2::DispatchHelper<caffe2::TensorTypes<FirstType, Types ...>, ExtraArgs ...>::call(Op*, const caffe2::TypeMeta&) [with Op = caffe2::RowWiseSparseAdagradOp<caffe2::CPUContext>; FirstType = long int; Types = {}; ExtraArgs = {}]'
/data/tseng/toolkit/pytorch/caffe2/core/operator.h:1302:1:   required from 'static bool caffe2::DispatchHelper<caffe2::TensorTypes<FirstType, Types ...>, ExtraArgs ...>::call(Op*, const caffe2::TypeMeta&) [with Op = caffe2::RowWiseSparseAdagradOp<caffe2::CPUContext>; FirstType = int; Types = {long int}; ExtraArgs = {}]'
/data/tseng/toolkit/pytorch/caffe2/core/operator.h:1302:1:   required from 'static bool caffe2::DispatchHelper<caffe2::TensorTypes<FirstType, Types ...>, ExtraArgs ...>::call(Op*, const caffe2::Tensor&) [with Op = caffe2::RowWiseSparseAdagradOp<caffe2::CPUContext>; FirstType = int; Types = {long int}; ExtraArgs = {}]'
/data/tseng/toolkit/pytorch/caffe2/sgd/adagrad_op.h:376:63:   required from 'bool caffe2::RowWiseSparseAdagradOp<Context>::RunOnDevice() [with Context = caffe2::CPUContext]'
/data/tseng/toolkit/pytorch/caffe2/sgd/adagrad_op.cc:215:1:   required from here
/data/tseng/toolkit/pytorch/caffe2/sgd/adagrad_op.h:434:26: error: no match for call to '(fbgemm::SparseAdaGradSignature<int>::Type {aka std::function<int(int, long unsigned int, float*, const float*, float*, const int*, float, float, float)>}) (long int&, int64_t, float*&, const float*&, float*&, const int32_t*, float&, const float& '
       num_rows_processed = kernel_i32_(
                          ^
In file included from /data/tseng/toolkit/pytorch/caffe2/core/operator.h:9:0,
                 from /data/tseng/toolkit/pytorch/caffe2/sgd/adagrad_op.h:3,
                 from /data/tseng/toolkit/pytorch/caffe2/sgd/adagrad_op.cc:1:
/usr/include/c++/5/functional:2262:5: note: candidate: _Res std::function<_Res(_ArgTypes ...)>::operator()(_ArgTypes ...) const [with _Res = int; _ArgTypes = {int, long unsigned int, float*, const float*, float*, const int*, float, float, float}]
     function<_Res(_ArgTypes...)>::
     ^
/usr/include/c++/5/functional:2262:5: note:   candidate expects 9 arguments, 8 provided
In file included from /data/tseng/toolkit/pytorch/caffe2/sgd/adagrad_op.cc:1:0:
/data/tseng/toolkit/pytorch/caffe2/sgd/adagrad_op.h:444:26: error: no match for call to '(fbgemm::SparseAdaGradSignature<long int>::Type {aka std::function<int(int, long unsigned int, float*, const float*, float*, const long int*, float, float, float)>}) (long int&, int64_t, float*&, const float*&, float*&, const long int*, float&, const float&)'
       num_rows_processed = kernel_i64_(
                          ^
In file included from /data/tseng/toolkit/pytorch/caffe2/core/operator.h:9:0,
                 from /data/tseng/toolkit/pytorch/caffe2/sgd/adagrad_op.h:3,
                 from /data/tseng/toolkit/pytorch/caffe2/sgd/adagrad_op.cc:1:
/usr/include/c++/5/functional:2262:5: note: candidate: _Res std::function<_Res(_ArgTypes ...)>::operator()(_ArgTypes ...) const [with _Res = int; _ArgTypes = {int, long unsigned int, float*, const float*, float*, const long int*, float, float, float}]
     function<_Res(_ArgTypes...)>::
     ^
/usr/include/c++/5/functional:2262:5: note:   candidate expects 9 arguments, 8 provided
caffe2/CMakeFiles/torch_cpu.dir/build.make:10984: recipe for target 'caffe2/CMakeFiles/torch_cpu.dir/sgd/adagrad_fused.cc.o' failed
make[2]: *** [caffe2/CMakeFiles/torch_cpu.dir/sgd/adagrad_fused.cc.o] Error 1
make[2]: *** Waiting for unfinished jobs....
caffe2/CMakeFiles/torch_cpu.dir/build.make:10997: recipe for target 'caffe2/CMakeFiles/torch_cpu.dir/sgd/adagrad_op.cc.o' failed
make[2]: *** [caffe2/CMakeFiles/torch_cpu.dir/sgd/adagrad_op.cc.o] Error 1
Archiving  objects                             > /data/tseng/toolkit/pytorch/build/nccl/obj/collectives/device/colldevice.a
Linking    libnccl.so.2.7.8                    > /data/tseng/toolkit/pytorch/build/nccl/lib/libnccl.so.2.7.8
Generating nccl.pc.in                          > /data/tseng/toolkit/pytorch/build/nccl/lib/pkgconfig/nccl.pc
Archiving  libnccl_static.a                    > /data/tseng/toolkit/pytorch/build/nccl/lib/libnccl_static.a
/data/tseng/toolkit/pytorch/third_party/nccl/nccl/src
[ 67%] No install step for 'nccl_external'
[ 67%] Completed 'nccl_external'
[ 67%] Built target nccl_external
[ 67%] Building NVCC (Device) object third_party/gloo/gloo/CMakeFiles/gloo_cuda.dir/gloo_cuda_generated_cuda.cu.o
[ 67%] Building NVCC (Device) object third_party/gloo/gloo/CMakeFiles/gloo_cuda.dir/nccl/gloo_cuda_generated_nccl.cu.o
[ 67%] Building NVCC (Device) object third_party/gloo/gloo/CMakeFiles/gloo_cuda.dir/gloo_cuda_generated_cuda_private.cu.o
CMakeFiles/Makefile2:9310: recipe for target 'caffe2/CMakeFiles/torch_cpu.dir/all' failed
make[1]: *** [caffe2/CMakeFiles/torch_cpu.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....
Scanning dependencies of target gloo_cuda
[ 68%] Building CXX object third_party/gloo/gloo/CMakeFiles/gloo_cuda.dir/cuda_allreduce_local.cc.o
[ 68%] Building CXX object third_party/gloo/gloo/CMakeFiles/gloo_cuda.dir/cuda_allreduce_ring.cc.o
[ 68%] Building CXX object third_party/gloo/gloo/CMakeFiles/gloo_cuda.dir/cuda_allreduce_bcube.cc.o
[ 68%] Building CXX object third_party/gloo/gloo/CMakeFiles/gloo_cuda.dir/cuda_allreduce_halving_doubling.cc.o
[ 68%] Building CXX object third_party/gloo/gloo/CMakeFiles/gloo_cuda.dir/cuda_broadcast_one_to_all.cc.o
[ 68%] Building CXX object third_party/gloo/gloo/CMakeFiles/gloo_cuda.dir/cuda_allreduce_ring_chunked.cc.o
[ 68%] Linking CXX static library ../../../lib/libgloo_cuda.a
[ 68%] Built target gloo_cuda
Makefile:159: recipe for target 'all' failed
make: *** [all] Error 2
Traceback (most recent call last):
  File "setup.py", line 717, in <module>
    build_deps()
  File "setup.py", line 313, in build_deps
    cmake=cmake)
  File "/data/tseng/toolkit/pytorch/tools/build_pytorch_libs.py", line 62, in build_caffe2
    cmake.build(my_env)
  File "/data/tseng/toolkit/pytorch/tools/setup_helpers/cmake.py", line 345, in build
    self.run(build_args, my_env)
  File "/data/tseng/toolkit/pytorch/tools/setup_helpers/cmake.py", line 141, in run
    check_call(command, cwd=self.build_dir, env=env)
  File "/opt/conda/lib/python3.6/subprocess.py", line 311, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['cmake', '--build', '.', '--target', 'install', '--config', 'Release', '--', '-j', '16']' returned non-zero exit status 2.
root@eb7398d8aac0:/data/tseng/toolkit/pytorch#

cc @malfet @seemethere @walterddr

[skyline75489]

Can you run collect_env.py and show us the output?

Please read the issue instruction and do not delete the issue template. Otherwise there's no way we can have enough information about the issue.

[CnybTseng]

Can you run collect_env.py and show us the output?

Please read the issue instruction and do not delete the issue template. Otherwise there's no way we can have enough information about the issue.

In PyTorch source directory:

python torch/utils/collect_env.py

Collecting environment information...
Traceback (most recent call last):
File "torch/utils/collect_env.py", line 429, in
main()
File "torch/utils/collect_env.py", line 424, in main
output = get_pretty_env_info()
File "torch/utils/collect_env.py", line 419, in get_pretty_env_info
return pretty_str(get_env_info())
File "torch/utils/collect_env.py", line 276, in get_env_info
if torch.version.hip is None: # cuda version
AttributeError: module 'torch.version' has no attribute 'hip'

[skyline75489]

No. Do not run the script under PyTorch source directory. The source code itself interferes with the collect_env.py. Copy the script somewhere else and run it.

[skyline75489]

Wait, you don't have any version of PyTorch installed? Guess collect_env.py is not ready if there is no PyTorch at all..

[zou3519]

We probably need more information to debug this, but it would be nice if collect_env.py worked without a PyTorch installation

[CnybTseng]

Wait, you don't have any version of PyTorch installed? Guess collect_env.py is not ready if there is no PyTorch at all..

I have installed PyTorch-1.2.0. Same error happened when I copy collect_env.py to another directory.

python collect_env.py

Collecting environment information...
Traceback (most recent call last):
File "collect_env.py", line 429, in
main()
File "collect_env.py", line 424, in main
output = get_pretty_env_info()
File "collect_env.py", line 419, in get_pretty_env_info
return pretty_str(get_env_info())
File "collect_env.py", line 276, in get_env_info
if torch.version.hip is None: # cuda version
AttributeError: module 'torch.version' has no attribute 'hip'

In my Python environment, import torch, and help(torch.version) will print:
Help on module torch.version in torch:

NAME
torch.version

DATA
cuda = '10.0.130'
debug = False
git_version = '8554416a199c4cec01c60c7015d8301d2bb39b64'

VERSION
1.2.0

FILE
/opt/conda/lib/python3.6/site-packages/torch/version.py

(END)

[CnybTseng]

We probably need more information to debug this, but it would be nice if collect_env.py worked without a PyTorch installation

I created a virtual environment without pytorch using: /opt/conda/bin/virtualenv ./tttest
Running collect_env.py will failed too:

python collect_env.py

Collecting environment information...
Traceback (most recent call last):
File "collect_env.py", line 429, in
main()
File "collect_env.py", line 424, in main
output = get_pretty_env_info()
File "collect_env.py", line 419, in get_pretty_env_info
return pretty_str(get_env_info())
File "collect_env.py", line 276, in get_env_info
if torch.version.hip is None: # cuda version
NameError: name 'torch' is not defined

[skyline75489]

OK, torch 1.2.0 is like kinda old. So I guess collect_env.py isn't really ready. But I could not understand NameError: name 'torch' is not defined. If import torch failed, then it shouldn't run the code who try to import torch.version.hip. What conda version are you using? My conda does not even have virtualenv.

[CnybTseng]

OK, torch 1.2.0 is like kinda old. So I guess collect_env.py isn't really ready. But I could not understand NameError: name 'torch' is not defined. If import torch failed, then it shouldn't run the code who try to import torch.version.hip. What conda version are you using? My conda does not even have virtualenv.

I'm using conda 4.9.2
When I use collect_env.py from pytorch master branch:

python collect_env.py

Collecting environment information...
PyTorch version: N/A
Is debug build: N/A
CUDA used to build PyTorch: N/A
ROCM used to build PyTorch: N/A

OS: Ubuntu 16.04.6 LTS (x86_64)
GCC version: (Ubuntu 5.4.0-6ubuntu1~16.04.11) 5.4.0 20160609
Clang version: Could not collect
CMake version: version 3.18.4

Python version: 3.6 (64-bit runtime)
Is CUDA available: N/A
CUDA runtime version: Could not collect
GPU models and configuration: GPU 0: GeForce RTX 2080 Ti
Nvidia driver version: 410.104
cuDNN version: /usr/lib/x86_64-linux-gnu/libcudnn.so.7.6.3
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip] Could not collect
[conda] Could not collect

By the way, I'm buiding pytorch from tag v1.7.0 instead of master branch.

[skyline75489]

OK I think the problem is that GCC 5.4.0 is a rather old version of GCC. AFAIK, the recommended version is somewhere around GCC 9.