Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cannot build Caffe2 while installing Pytorch on Raspbian (buster) #22564

Open
nlhkh opened this issue Jul 6, 2019 · 16 comments

Comments

@nlhkh
Copy link

commented Jul 6, 2019

馃悰 Bug

Cannot build caffe2 while installing Pytorch on Raspberry Pi 3B+ (Raspbian:buster). Error: undefined reference to `__atomic_fetch_add_8'.

To Reproduce

Steps to reproduce the behavior:

  1. Get pytorch git clone --recursive https://github.com/pytorch/pytorch

  2. Set environment variables

export NO_CUDA=1
export NO_DISTRIBUTED=1
export NO_MKLDNN=1 
export NO_NNPACK=1
export NO_QNNPACK=1
  1. Run build python3 setup.py build

Error message:

[ 47%] Linking CXX executable ../../../bin/protoc
[ 47%] Built target sleefneon32
[ 47%] Built target gmock_main
[ 47%] Built target sleef
/usr/bin/ld: ../../../lib/libprotobuf.a(arena.cc.o): in function `google::protobuf::internal::ArenaImpl::Init()':
arena.cc:(.text+0x24): undefined reference to `__atomic_fetch_add_8'
collect2: error: ld returned 1 exit status
make[2]: *** [third_party/protobuf/cmake/CMakeFiles/protoc.dir/build.make:87: bin/protoc] Error 1
make[1]: *** [CMakeFiles/Makefile2:241: third_party/protobuf/cmake/CMakeFiles/protoc.dir/all] Error 2
make: *** [Makefile:141: all] Error 2
Traceback (most recent call last):
  File "setup.py", line 749, in <module>
    build_deps()
  File "setup.py", line 323, in build_deps
    cmake=cmake)
  File "/home/pi/pytorch_install/pytorch/tools/build_pytorch_libs.py", line 64, in build_caffe2
    cmake.build(my_env)
  File "/home/pi/pytorch_install/pytorch/tools/setup_helpers/cmake.py", line 345, in build
    self.run(build_args, my_env)
  File "/home/pi/pytorch_install/pytorch/tools/setup_helpers/cmake.py", line 107, in run
    check_call(command, cwd=self.build_dir, env=env)
  File "/usr/lib/python3.7/subprocess.py", line 347, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['cmake', '--build', '.', '--target', 'install', '--config', 'Release', '--', '-j', '4']' returned non-zero exit status 2.

Expected behavior

Build should complete successfully

Environment

  • Collecting environment information...

  • PyTorch version: N/A

  • Is debug build: N/A

  • CUDA used to build PyTorch: N/A

  • OS: Raspbian GNU/Linux 10 (buster)

  • GCC version: (Raspbian 8.3.0-6+rpi1) 8.3.0

  • CMake version: version 3.13.4

  • Python version: 3.7

  • Is CUDA available: N/A

  • CUDA runtime version: Could not collect

  • GPU models and configuration: Could not collect

  • Nvidia driver version: Could not collect

  • cuDNN version: Could not collect

Versions of relevant libraries:

  • [pip3] numpy==1.16.2
  • [pip3] torchfile==0.1.0
  • [conda] Could not collect

Additional context

@xuhdev

This comment has been minimized.

Copy link
Member

commented Jul 6, 2019

Are you using clang for building? It is suggested that switching to gcc or adding the -stdlib=libstdc++ flag to clang (you can do this by editing build/CMakeCache.txt and adding that to CMAKE_CXX_FLAGS:STRING=) might resolve this issue:

https://stackoverflow.com/questions/28920489/how-come-stdatomicdouble-isnt-implemented-when-compiling-with-clang

@xuhdev xuhdev added the module: build label Jul 6, 2019

@nlhkh

This comment has been minimized.

Copy link
Author

commented Jul 6, 2019

Hi @xuhdev, I did as you said CMAKE_CXX_FLAGS:STRING=-stdlib=libstdc++, but it says that -stdlib=libstdc++ is an undefined symbol.

c++: error: unrecognized command line option 鈥-stdlib=libstdc++鈥; did you mean 鈥-static-libstdc++鈥?

I am not familiar with the CMake build tool though.

Should I change something in the root CMakeList.txt instead?

@xuhdev

This comment has been minimized.

Copy link
Member

commented Jul 7, 2019

You shouldn't need to modify CMakeLists.txt. This likely means you are using gcc not clang. Just to confirm, what is the output of c++ -v on your system?

@nlhkh

This comment has been minimized.

Copy link
Author

commented Jul 7, 2019

I tried both these without success. I put them in the root CMakeList.txt file.
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -latomic") # same error message as before
and
set(CMAKE_CXX_LINK_FLAGS "${CMAKE_CXX_LINK_FLAGS} -latomic") # failed in build_deps

[ 44%] Linking CXX executable ../../bin/c10_intrusive_ptr_test
[ 44%] Built target c10_intrusive_ptr_test
make: *** [Makefile:141: all] Error 2
Traceback (most recent call last):
  File "setup.py", line 725, in <module>
    build_deps()
  File "setup.py", line 285, in build_deps
    build_dir='build')
  File "/home/pi/pytorch_install/pytorch/tools/build_pytorch_libs.py", line 281, in build_caffe2
    check_call(['make', '-j', str(max_jobs), 'install'], cwd=build_dir, env=my_env)
  File "/usr/lib/python3.7/subprocess.py", line 347, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['make', '-j', '4', 'install']' returned non-zero exit status 2.

Please be reminded that I am installing Pytorch on a Raspberry Pi.

@nlhkh

This comment has been minimized.

Copy link
Author

commented Jul 7, 2019

@xuhdev Here is the output of c++ -v

Using built-in specs.
COLLECT_GCC=c++
COLLECT_LTO_WRAPPER=/usr/lib/gcc/arm-linux-gnueabihf/8/lto-wrapper
Target: arm-linux-gnueabihf
Configured with: ../src/configure -v --with-pkgversion='Raspbian 8.3.0-6+rpi1' --with-bugurl=file:///usr/share/doc/gcc-8/README.Bugs --enable-languages=c,ada,c++,go,d,fortran,objc,obj-c++ --prefix=/usr --with-gcc-major-version-only --program-suffix=-8 --program-prefix=arm-linux-gnueabihf- --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --libdir=/usr/lib --enable-nls --enable-bootstrap --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --with-default-libstdcxx-abi=new --enable-gnu-unique-object --disable-libitm --disable-libquadmath --disable-libquadmath-support --enable-plugin --with-system-zlib --with-target-system-zlib --enable-objc-gc=auto --enable-multiarch --disable-sjlj-exceptions --with-arch=armv6 --with-fpu=vfp --with-float=hard --disable-werror --enable-checking=release --build=arm-linux-gnueabihf --host=arm-linux-gnueabihf --target=arm-linux-gnueabihf
Thread model: posix
gcc version 8.3.0 (Raspbian 8.3.0-6+rpi1)
@xuhdev

This comment has been minimized.

Copy link
Member

commented Jul 7, 2019

Could you export MAX_JOBS=1 when you try to produce the build error? It looks like your second latest comment has the real error message missed.

@nlhkh

This comment has been minimized.

Copy link
Author

commented Jul 7, 2019

With export MAX_JOBS=1 it failed for the same reason

[ 44%] Linking CXX executable ../../../bin/protoc
/usr/bin/ld: ../../../lib/libprotobuf.a(arena.cc.o): in function `google::protobuf::internal::ArenaImpl::Init()':
arena.cc:(.text+0x24): undefined reference to `__atomic_fetch_add_8'
collect2: error: ld returned 1 exit status
make[2]: *** [third_party/protobuf/cmake/CMakeFiles/protoc.dir/build.make:87: bin/protoc] Error 1
make[1]: *** [CMakeFiles/Makefile2:241: third_party/protobuf/cmake/CMakeFiles/protoc.dir/all] Error 2
make: *** [Makefile:141: all] Error 2
Traceback (most recent call last):
  File "setup.py", line 725, in <module>
    build_deps()
  File "setup.py", line 285, in build_deps
    build_dir='build')
  File "/home/pi/pytorch_install/pytorch/tools/build_pytorch_libs.py", line 281, in build_caffe2
    check_call(['make', '-j', str(max_jobs), 'install'], cwd=build_dir, env=my_env)
  File "/usr/lib/python3.7/subprocess.py", line 347, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['make', '-j', '1', 'install']' returned non-zero exit status 2.
@xuhdev

This comment has been minimized.

Copy link
Member

commented Jul 7, 2019

OK. Could you run:

cd build
make -j 1 VERBOSE=1

and post the output?

@nlhkh

This comment has been minimized.

Copy link
Author

commented Jul 7, 2019

Here you are

/usr/bin/cmake -S/home/pi/pytorch_install/pytorch -B/home/pi/pytorch_install/pytorch/build --check-build-system CMakeFiles/Makefile.cmake 0
/usr/bin/cmake -E cmake_progress_start /home/pi/pytorch_install/pytorch/build/CMakeFiles /home/pi/pytorch_install/pytorch/build/CMakeFiles/progress.marks
make -f CMakeFiles/Makefile2 all
make[1]: Entering directory '/home/pi/pytorch_install/pytorch/build'
make -f third_party/protobuf/cmake/CMakeFiles/libprotobuf.dir/build.make third_party/protobuf/cmake/CMakeFiles/libprotobuf.dir/depend
make[2]: Entering directory '/home/pi/pytorch_install/pytorch/build'
cd /home/pi/pytorch_install/pytorch/build && /usr/bin/cmake -E cmake_depends "Unix Makefiles" /home/pi/pytorch_install/pytorch /home/pi/pytorch_install/pytorch/third_party/protobuf/cmake /home/pi/pytorch_install/pytorch/build /home/pi/pytorch_install/pytorch/build/third_party/protobuf/cmake /home/pi/pytorch_install/pytorch/build/third_party/protobuf/cmake/CMakeFiles/libprotobuf.dir/DependInfo.cmake --color=
make[2]: Leaving directory '/home/pi/pytorch_install/pytorch/build'
make -f third_party/protobuf/cmake/CMakeFiles/libprotobuf.dir/build.make third_party/protobuf/cmake/CMakeFiles/libprotobuf.dir/build
make[2]: Entering directory '/home/pi/pytorch_install/pytorch/build'
make[2]: Nothing to be done for 'third_party/protobuf/cmake/CMakeFiles/libprotobuf.dir/build'.
make[2]: Leaving directory '/home/pi/pytorch_install/pytorch/build'
[ 44%] Built target libprotobuf
make -f third_party/protobuf/cmake/CMakeFiles/libprotobuf-lite.dir/build.make third_party/protobuf/cmake/CMakeFiles/libprotobuf-lite.dir/depend
make[2]: Entering directory '/home/pi/pytorch_install/pytorch/build'
cd /home/pi/pytorch_install/pytorch/build && /usr/bin/cmake -E cmake_depends "Unix Makefiles" /home/pi/pytorch_install/pytorch /home/pi/pytorch_install/pytorch/third_party/protobuf/cmake /home/pi/pytorch_install/pytorch/build /home/pi/pytorch_install/pytorch/build/third_party/protobuf/cmake /home/pi/pytorch_install/pytorch/build/third_party/protobuf/cmake/CMakeFiles/libprotobuf-lite.dir/DependInfo.cmake --color=
make[2]: Leaving directory '/home/pi/pytorch_install/pytorch/build'
make -f third_party/protobuf/cmake/CMakeFiles/libprotobuf-lite.dir/build.make third_party/protobuf/cmake/CMakeFiles/libprotobuf-lite.dir/build
make[2]: Entering directory '/home/pi/pytorch_install/pytorch/build'
make[2]: Nothing to be done for 'third_party/protobuf/cmake/CMakeFiles/libprotobuf-lite.dir/build'.
make[2]: Leaving directory '/home/pi/pytorch_install/pytorch/build'
[ 44%] Built target libprotobuf-lite
make -f third_party/protobuf/cmake/CMakeFiles/libprotoc.dir/build.make third_party/protobuf/cmake/CMakeFiles/libprotoc.dir/depend
make[2]: Entering directory '/home/pi/pytorch_install/pytorch/build'
cd /home/pi/pytorch_install/pytorch/build && /usr/bin/cmake -E cmake_depends "Unix Makefiles" /home/pi/pytorch_install/pytorch /home/pi/pytorch_install/pytorch/third_party/protobuf/cmake /home/pi/pytorch_install/pytorch/build /home/pi/pytorch_install/pytorch/build/third_party/protobuf/cmake /home/pi/pytorch_install/pytorch/build/third_party/protobuf/cmake/CMakeFiles/libprotoc.dir/DependInfo.cmake --color=
make[2]: Leaving directory '/home/pi/pytorch_install/pytorch/build'
make -f third_party/protobuf/cmake/CMakeFiles/libprotoc.dir/build.make third_party/protobuf/cmake/CMakeFiles/libprotoc.dir/build
make[2]: Entering directory '/home/pi/pytorch_install/pytorch/build'
make[2]: Nothing to be done for 'third_party/protobuf/cmake/CMakeFiles/libprotoc.dir/build'.
make[2]: Leaving directory '/home/pi/pytorch_install/pytorch/build'
[ 44%] Built target libprotoc
make -f third_party/protobuf/cmake/CMakeFiles/protoc.dir/build.make third_party/protobuf/cmake/CMakeFiles/protoc.dir/depend
make[2]: Entering directory '/home/pi/pytorch_install/pytorch/build'
cd /home/pi/pytorch_install/pytorch/build && /usr/bin/cmake -E cmake_depends "Unix Makefiles" /home/pi/pytorch_install/pytorch /home/pi/pytorch_install/pytorch/third_party/protobuf/cmake /home/pi/pytorch_install/pytorch/build /home/pi/pytorch_install/pytorch/build/third_party/protobuf/cmake /home/pi/pytorch_install/pytorch/build/third_party/protobuf/cmake/CMakeFiles/protoc.dir/DependInfo.cmake --color=
make[2]: Leaving directory '/home/pi/pytorch_install/pytorch/build'
make -f third_party/protobuf/cmake/CMakeFiles/protoc.dir/build.make third_party/protobuf/cmake/CMakeFiles/protoc.dir/build
make[2]: Entering directory '/home/pi/pytorch_install/pytorch/build'
[ 44%] Linking CXX executable ../../../bin/protoc
cd /home/pi/pytorch_install/pytorch/build/third_party/protobuf/cmake && /usr/bin/cmake -E cmake_link_script CMakeFiles/protoc.dir/link.txt --verbose=1
/usr/bin/c++    -Wno-deprecated -fvisibility-inlines-hidden -fvisibility=hidden -fvisibility-inlines-hidden -O3 -DNDEBUG   -rdynamic -rdynamic CMakeFiles/protoc.dir/__/src/google/protobuf/compiler/main.cc.o  -o ../../../bin/protoc -Wl,-rpath,::::::: ../../../lib/libprotobuf.a ../../../lib/libprotoc.a ../../../lib/libprotobuf.a -pthread
/usr/bin/ld: ../../../lib/libprotobuf.a(arena.cc.o): in function `google::protobuf::internal::ArenaImpl::Init()':
arena.cc:(.text+0x24): undefined reference to `__atomic_fetch_add_8'
collect2: error: ld returned 1 exit status
make[2]: *** [third_party/protobuf/cmake/CMakeFiles/protoc.dir/build.make:87: bin/protoc] Error 1
make[2]: Leaving directory '/home/pi/pytorch_install/pytorch/build'
make[1]: *** [CMakeFiles/Makefile2:241: third_party/protobuf/cmake/CMakeFiles/protoc.dir/all] Error 2
make[1]: Leaving directory '/home/pi/pytorch_install/pytorch/build'
make: *** [Makefile:141: all] Error 2
@xuhdev

This comment has been minimized.

Copy link
Member

commented Jul 7, 2019

It looks like this is a bug in protobuf and it has been fixed a few days ago: protocolbuffers/protobuf@55ed1d4

Could you run the following and try to build again?

git submodule update --remote third_party/protobuf
@xuhdev

This comment has been minimized.

Copy link
Member

commented Jul 8, 2019

Exactly the same error: protocolbuffers/protobuf#6337

@zou3519 zou3519 added the triaged label Jul 8, 2019

@nlhkh

This comment has been minimized.

Copy link
Author

commented Jul 8, 2019

It works! Thanks @xuhdev.
The compile time on a Raspberry Pi 3B+ was almost 12 hours though, so it took me a while to reply.

@xuhdev

This comment has been minimized.

Copy link
Member

commented Jul 8, 2019

@nlhkh To speed up, you can also pass BUILD_TEST=0 to the build command

xuhdev added a commit to xuhdev/pytorch that referenced this issue Jul 8, 2019

Update protobuf submodule.
A build issue on Raspberry Pi that affects PyTorch was fixed only
recently: protocolbuffers/protobuf@55ed1d4

Close pytorch#22564

@xuhdev xuhdev referenced a pull request that will close this issue Jul 8, 2019

Open

[WIP] Update protobuf submodule. #22595

@sautax

This comment has been minimized.

Copy link

commented Jul 9, 2019

i have a question @xuhdev , when you say to pass BUILD_TEST=0 to the build command, do you mean export BUILD_TEST=0 ?

@xuhdev

This comment has been minimized.

Copy link
Member

commented Jul 9, 2019

Yes

xuhdev added a commit to xuhdev/pytorch that referenced this issue Jul 10, 2019

Update protobuf submodule.
A build issue on Raspberry Pi that affects PyTorch was fixed only
recently: protocolbuffers/protobuf@55ed1d4

Close pytorch#22564

xuhdev added a commit to xuhdev/pytorch that referenced this issue Jul 10, 2019

Update protobuf submodule.
A build issue on Raspberry Pi that affects PyTorch was fixed only
recently:

- protocolbuffers/protobuf@55ed1d4
- protocolbuffers/protobuf@56e7bdf

Close pytorch#22564
@andcarnivorous

This comment has been minimized.

Copy link

commented Aug 8, 2019

Thank you

It looks like this is a bug in protobuf and it has been fixed a few days ago: protocolbuffers/protobuf@55ed1d4

Could you run the following and try to build again?

git submodule update --remote third_party/protobuf

This fixed it for me as well.

xuhdev added a commit to xuhdev/pytorch that referenced this issue Aug 8, 2019

Update protobuf submodule.
A build issue on Raspberry Pi that affects PyTorch was fixed only
recently:

- protocolbuffers/protobuf@55ed1d4
- protocolbuffers/protobuf@56e7bdf

Close pytorch#22564

xuhdev added a commit to xuhdev/pytorch that referenced this issue Aug 8, 2019

Update protobuf submodule.
A build issue on Raspberry Pi that affects PyTorch was fixed only
recently:

- protocolbuffers/protobuf@55ed1d4
- protocolbuffers/protobuf@56e7bdf

Close pytorch#22564
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
5 participants
You can鈥檛 perform that action at this time.