Linking CXX executable onnxruntime_test_all fails when cross-compiling onnxruntime for arm

[drux007]

Describe the bug
When I try to build onnxruntime for arm by cross compiling on Linux using arm-linux-gnueabihf-gcc-7 toolchain, Linking CXX executable onnxruntime_test_all fails.

Urgency
We use onnxruntime for deploying our models on various different platforms and also we need to add ARM platform as soon as possible.

System information

• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Linux Ubuntu 18.04
• ONNX Runtime installed from (source or binary): source
• ONNX Runtime version: 1.8.0
• Python version: 3.6.9
• GCC/Compiler version (if compiling from source):7.5.0

I tried to build it using docker image conanio/gcc7-armv7:

• OS Platform: Linux
• ONNX Runtime installed from (source or binary): source
• ONNX Runtime version: 1.8.0
• GCC/Compiler: arm-linux-gnueabihf-gcc-7
• GCC/Compiler version (if compiling from source):7.2.0

To Reproduce
The easiest is to simply run it inside the following docker container:

docker run -it conanio/gcc7-armv7 /bin/bash

Download the source code:

git clone --single-branch --branch v1.8.0 --recursive https://github.com/Microsoft/onnxruntime

Download pre-compiled protoc version 3.16.0 inside docker container from here, as described in the official instructions and unzip it after downloading.. The version must match the one onnxruntime is using. Currently that is 3.16.0 (see here).

Start build process with:

./build.sh --path_to_protoc_exe=/home/conan/protoc-3.16.0-linux-x86_64/bin/protoc --cmake_extra_defines="CMAKE_C_COMPILER=/usr/bin/arm-linux-gnueabihf-gcc-7" --cmake_extra_defines="CMAKE_CXX_COMPILER=/usr/bin/arm-linux-gnueabihf-g++-7" --parallel --config Release --build_dir build --skip_tests

Note: Adapt the path to pre-compiled-protoc.

Expected behavior
Expected would be that it builds successfully, but it fails at Linking CXX executable onnxruntime_test_all with:

Linking CXX executable onnxruntime_test_all
CMakeFiles/onnxruntime_test_all.dir/home/conan/onnxruntime/onnxruntime/test/providers/cpu/controlflow/loop_test.cc.o: In function `std::__atomic_futex_unsigned<2147483648u>::_M_store_notify_all(unsigned int, std::memory_order)':
/usr/arm-linux-gnueabihf/include/c++/7/bits/atomic_futex.h:208: undefined reference to `std::__atomic_futex_unsigned_base::_M_futex_notify_all(unsigned int*)'
/usr/arm-linux-gnueabihf/include/c++/7/bits/atomic_futex.h:208: undefined reference to `std::__atomic_futex_unsigned_base::_M_futex_notify_all(unsigned int*)'
CMakeFiles/onnxruntime_test_all.dir/home/conan/onnxruntime/onnxruntime/test/providers/cpu/controlflow/loop_test.cc.o: In function `std::__atomic_futex_unsigned<2147483648u>::_M_load_and_test_until(unsigned int, unsigned int, bool, std::memory_order, bool, std::chrono::duration<long long, std::ratio<1ll, 1ll> >, std::chrono::duration<long long, std::ratio<1ll, 1000000000ll> >)':
/usr/arm-linux-gnueabihf/include/c++/7/bits/atomic_futex.h:102: undefined reference to `std::__atomic_futex_unsigned_base::_M_futex_wait_until(unsigned int*, unsigned int, bool, std::chrono::duration<long long, std::ratio<1ll, 1ll> >, std::chrono::duration<long long, std::ratio<1ll, 1000000000ll> >)'
collect2: error: ld returned 1 exit status
CMakeFiles/onnxruntime_test_all.dir/build.make:3589: recipe for target 'onnxruntime_test_all' failed
make[2]: *** [onnxruntime_test_all] Error 1
CMakeFiles/Makefile2:1072: recipe for target 'CMakeFiles/onnxruntime_test_all.dir/all' failed
make[1]: *** [CMakeFiles/onnxruntime_test_all.dir/all] Error 2
Makefile:181: recipe for target 'all' failed
make: *** [all] Error 2
Traceback (most recent call last):
  File "/home/conan/onnxruntime/tools/ci_build/build.py", line 2152, in <module>
    sys.exit(main())
  File "/home/conan/onnxruntime/tools/ci_build/build.py", line 2076, in main
    build_targets(args, cmake_path, build_dir, configs, num_parallel_jobs, args.target)
  File "/home/conan/onnxruntime/tools/ci_build/build.py", line 1067, in build_targets
    run_subprocess(cmd_args, env=env)
  File "/home/conan/onnxruntime/tools/ci_build/build.py", line 581, in run_subprocess
    return run(*args, cwd=cwd, capture_stdout=capture_stdout, shell=shell, env=my_env)
  File "/home/conan/onnxruntime/tools/python/util/run.py", line 44, in run
    env=env, shell=shell)
  File "/opt/pyenv/versions/3.7.5/lib/python3.7/subprocess.py", line 512, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['/usr/bin/cmake', '--build', 'build/Debug', '--config', 'Debug', '--', '-j8']' returned non-zero exit status 2.

[drux007]

I also tried to build it by using cmake directly:

cd onnxruntime
cmake -DCMAKE_CXX_COMPILER=/usr/bin/arm-linux-gnueabihf-g++-7 -DCMAKE_C_COMPILER=/usr/bin/arm-linux-gnueabihf-gcc-7 -DONNX_CUSTOM_PROTOC_EXECUTABLE=/home/conan/protoc-3.16.0-linux-x86_64/bin/protoc ./cmake
make

but again it fails at Linking CXX executable onnxruntime_test_all.

[snnn]

Please open CMakeCache.txt and search onnxruntime_LINK_LIBATOMIC, then post the value here.

[snnn]

As a quick fix, you can
Open CMakeLists.txt, change

set(onnxruntime_LINK_LIBATOMIC false)
# We don't need to link libatomic on iOS
if (NOT MSVC AND NOT (CMAKE_SYSTEM_NAME STREQUAL "iOS"))
  include(CheckCXXSourceCompiles)
  set(OLD_CMAKE_REQUIRED_FLAGS ${CMAKE_REQUIRED_FLAGS})
  set(CMAKE_REQUIRED_FLAGS ${CMAKE_REQUIRED_FLAGS} -std=c++11)
  check_cxx_source_compiles("
    #include <atomic>
    int main() {
      return std::atomic<int64_t>{};
    }
  " onnxruntime_HAVE_BUILTIN_ATOMICS)
  if (NOT onnxruntime_HAVE_BUILTIN_ATOMICS)
    set(onnxruntime_LINK_LIBATOMIC true)
  endif ()
  set(CMAKE_REQUIRED_FLAGS ${OLD_CMAKE_REQUIRED_FLAGS})
endif ()

to

set(onnxruntime_LINK_LIBATOMIC true)

[drux007]

Thank you @snnn for your quick answer!

Please open CMakeCache.txt and search onnxruntime_LINK_LIBATOMIC, then post the value here.

There is no onnxruntime_LINK_LIBATOMIC in CMakeCache.txt. Neither in the one generated using build.sh nor the one with cmake.

[drux007]

As a quick fix, you can
Open CMakeLists.txt, change

set(onnxruntime_LINK_LIBATOMIC false)
# We don't need to link libatomic on iOS
if (NOT MSVC AND NOT (CMAKE_SYSTEM_NAME STREQUAL "iOS"))
  include(CheckCXXSourceCompiles)
  set(OLD_CMAKE_REQUIRED_FLAGS ${CMAKE_REQUIRED_FLAGS})
  set(CMAKE_REQUIRED_FLAGS ${CMAKE_REQUIRED_FLAGS} -std=c++11)
  check_cxx_source_compiles("
    #include <atomic>
    int main() {
      return std::atomic<int64_t>{};
    }
  " onnxruntime_HAVE_BUILTIN_ATOMICS)
  if (NOT onnxruntime_HAVE_BUILTIN_ATOMICS)
    set(onnxruntime_LINK_LIBATOMIC true)
  endif ()
  set(CMAKE_REQUIRED_FLAGS ${OLD_CMAKE_REQUIRED_FLAGS})
endif ()

to

set(onnxruntime_LINK_LIBATOMIC true)

I tried it out, but it still fails with:

[ 72%] Linking CXX executable onnxruntime_test_all
CMakeFiles/onnxruntime_test_all.dir/home/conan/onnxruntime/onnxruntime/test/providers/cpu/controlflow/loop_test.cc.o: In function `std::__atomic_futex_unsigned<2147483648u>::_M_store_notify_all(unsigned int, std::memory_order)':
/usr/arm-linux-gnueabihf/include/c++/7/bits/atomic_futex.h:208: undefined reference to `std::__atomic_futex_unsigned_base::_M_futex_notify_all(unsigned int*)'
/usr/arm-linux-gnueabihf/include/c++/7/bits/atomic_futex.h:208: undefined reference to `std::__atomic_futex_unsigned_base::_M_futex_notify_all(unsigned int*)'
CMakeFiles/onnxruntime_test_all.dir/home/conan/onnxruntime/onnxruntime/test/providers/cpu/controlflow/loop_test.cc.o: In function `std::__atomic_futex_unsigned<2147483648u>::_M_load_and_test_until(unsigned int, unsigned int, bool, std::memory_order, bool, std::chrono::duration<long long, std::ratio<1ll, 1ll> >, std::chrono::duration<long long, std::ratio<1ll, 1000000000ll> >)':
/usr/arm-linux-gnueabihf/include/c++/7/bits/atomic_futex.h:102: undefined reference to `std::__atomic_futex_unsigned_base::_M_futex_wait_until(unsigned int*, unsigned int, bool, std::chrono::duration<long long, std::ratio<1ll, 1ll> >, std::chrono::duration<long long, std::ratio<1ll, 1000000000ll> >)'
collect2: error: ld returned 1 exit status
CMakeFiles/onnxruntime_test_all.dir/build.make:3589: recipe for target 'onnxruntime_test_all' failed
make[2]: *** [onnxruntime_test_all] Error 1
CMakeFiles/Makefile2:1072: recipe for target 'CMakeFiles/onnxruntime_test_all.dir/all' failed
make[1]: *** [CMakeFiles/onnxruntime_test_all.dir/all] Error 2
Makefile:181: recipe for target 'all' failed
make: *** [all] Error 2

[snnn]

When you run make, could you please do it with extra args like:

make VERBOSE=1

Then post the failed linking command here.

[drux007]

When you run make, could you please do it with extra args like:

make VERBOSE=1

Then post the failed linking command here.

@snnn, please find below the build log when specifying VERBOSE=1, as you suggested:

[ 72%] Linking CXX executable onnxruntime_test_all
/usr/bin/cmake -E copy_directory /home/conan/onnxruntime/onnxruntime/test/testdata /home/conan/onnxruntime/testdata
/usr/bin/cmake -E copy_directory /home/conan/onnxruntime/samples /home/conan/onnxruntime/samples
/usr/bin/cmake -E cmake_link_script CMakeFiles/onnxruntime_test_all.dir/link.txt --verbose=1
/usr/bin/arm-linux-gnueabihf-g++-7  -ffunction-sections -fdata-sections -mfpu=neon -mfloat-abi=hard -O2 -g -DNDEBUG -DGSL_UNENFORCED_ON_CONTRACT_VIOLATION  -Wl,-rpath='$ORIGIN' CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/common/denormal_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/common/logging/logging_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/common/logging/sinks_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/common/path_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/common/string_utils_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/common/tensor_op_test_utils.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/common/utf8_util_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/ir/common_data_structure_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/ir/graph_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/ir/graph_viewer_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/ir/onnx_model_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/ir/op_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/ir/schema_registry_manager_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/ir/utils_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/optimizer/cse_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/optimizer/free_dimension_override_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/optimizer/graph_transform_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/optimizer/graph_transform_test_builder.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/optimizer/graph_transform_utils_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/optimizer/initializer_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/optimizer/nchwc_optimizer_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/optimizer/nhwc_transformer_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/optimizer/optimizer_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/optimizer/qdq_transformer_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/optimizer/rule_based_graph_transformer_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/framework/TestAllocatorManager.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/framework/allocation_planner_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/framework/allocator_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/framework/bfc_arena_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/framework/data_types_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/framework/distance_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/framework/dummy_allocator.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/framework/dummy_provider.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/framework/endian_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/framework/execution_frame_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/framework/execution_provider_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/framework/float_16_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/framework/inference_session_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/framework/insert_cast_transformer_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/framework/kernel_def_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/framework/kernel_registry_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/framework/local_kernel_registry_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/framework/math_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/framework/mem_pattern_planner_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/framework/memcpy_transformer_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/framework/opaque_kernels_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/framework/ort_model_only_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/framework/parallel_executor_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/framework/provider_options_utils_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/framework/random_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/framework/save_model_with_external_initializers.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/framework/session_state_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/framework/shape_inference_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/framework/sparse_kernels_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/framework/tensor_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/framework/tensorutils_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/framework/test_tensor_loader.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/framework/test_utils.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/platform/barrier_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/platform/env_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/platform/file_io_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/platform/path_lib_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/platform/scoped_resource_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/platform/threadpool_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/contrib_ops/activation_op_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/contrib_ops/attention_lstm_op_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/contrib_ops/attention_op_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/contrib_ops/bias_dropout_op_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/contrib_ops/bias_softmax_op_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/contrib_ops/cdist_op_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/contrib_ops/conv_transpose_with_dynamic_pads_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/contrib_ops/crop_and_resize_op_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/contrib_ops/crop_op_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/contrib_ops/dynamic_quantize_matmul_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/contrib_ops/dynamic_slice_op_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/contrib_ops/element_wise_ops_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/contrib_ops/embedlayernorm_op_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/contrib_ops/expand_dims_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/contrib_ops/fastgelu_op_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/contrib_ops/fft_op_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/contrib_ops/function_ops_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/contrib_ops/function_test_util.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/contrib_ops/fused_conv_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/contrib_ops/fused_matmul_op_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/contrib_ops/inverse_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/contrib_ops/layer_norm_op_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/contrib_ops/layer_norm_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/contrib_ops/longformer_attention_op_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/contrib_ops/matmul_integer16_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/contrib_ops/matmul_integer_to_float_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/contrib_ops/maxpool_mask_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/contrib_ops/murmur_hash3_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/contrib_ops/nhwc_maxpool_op_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/contrib_ops/op_reg_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/contrib_ops/qlinear_binary_op_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/contrib_ops/qlinear_concat_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/contrib_ops/qlinear_global_average_pool_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/contrib_ops/qlinear_lookup_table_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/contrib_ops/qlinear_pool_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/contrib_ops/quantize_attention_op_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/contrib_ops/quantize_lstm_op_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/contrib_ops/quantize_ops_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/contrib_ops/sample_op_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/contrib_ops/signal_ops_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/contrib_ops/skiplayernorm_op_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/contrib_ops/tensor_op_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/contrib_ops/tokenizer_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/contrib_ops/torch_embedding_shape_inference_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/contrib_ops/trilu_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/contrib_ops/unique_op_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/contrib_ops/word_conv_embedding_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/opaque_api/test_opaque_api.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/providers/compare_provider_test_utils.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/providers/get_execution_providers_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/providers/kernel_def_hash_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/providers/memcpy_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/providers/op_kernel_type_control_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/providers/provider_test_utils.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/providers/cpu/activation/activation_op_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/providers/cpu/controlflow/if_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/providers/cpu/controlflow/loop_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/providers/cpu/controlflow/scan_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/providers/cpu/cpu_execution_provider_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/providers/cpu/generator/constant_of_shape_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/providers/cpu/generator/random_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/providers/cpu/generator/range_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/providers/cpu/math/clip_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/providers/cpu/math/cumsum_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/providers/cpu/math/det_op_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/providers/cpu/math/einsum_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/providers/cpu/math/element_wise_ops_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/providers/cpu/math/gemm_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/providers/cpu/math/hardmax_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/providers/cpu/math/logsoftmax_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/providers/cpu/math/matmul_integer_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/providers/cpu/math/matmul_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/providers/cpu/math/quantize_linear_matmul_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/providers/cpu/math/round_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/providers/cpu/math/sign_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/providers/cpu/math/softmax_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/providers/cpu/math/topk_op_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/providers/cpu/ml/array_feature_extractor_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/providers/cpu/ml/binarizer_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/providers/cpu/ml/cast_map_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/providers/cpu/ml/category_mapper_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/providers/cpu/ml/dictvectorizer_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/providers/cpu/ml/feature_vectorizer_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/providers/cpu/ml/imputer_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/providers/cpu/ml/label_encoder_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/providers/cpu/ml/linearclassifer_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/providers/cpu/ml/linearregressor_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/providers/cpu/ml/normalizer_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/providers/cpu/ml/onehotencoder_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/providers/cpu/ml/scaler_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/providers/cpu/ml/svmclassifier_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/providers/cpu/ml/svmregressor_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/providers/cpu/ml/tree_ensembler_classifier_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/providers/cpu/ml/treeregressor_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/providers/cpu/ml/write_scores_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/providers/cpu/ml/zipmap_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/providers/cpu/model_tests.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/providers/cpu/nn/batch_norm_op_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/providers/cpu/nn/conv_integer_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/providers/cpu/nn/conv_op_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/providers/cpu/nn/conv_transpose_op_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/providers/cpu/nn/dropout_op_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/providers/cpu/nn/flatten_op_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/providers/cpu/nn/instance_norm_op_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/providers/cpu/nn/lp_norm_op_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/providers/cpu/nn/lrn_op_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/providers/cpu/nn/max_roi_pool_op_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/providers/cpu/nn/pool_op_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/providers/cpu/nn/qlinearconv_op_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/providers/cpu/nn/shrink_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/providers/cpu/nn/string_normalizer_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/providers/cpu/nn/tfidfvectorizer_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/providers/cpu/nn/unpool_op_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/providers/cpu/object_detection/non_max_suppression_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/providers/cpu/object_detection/roialign_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/providers/cpu/reduction/reduction_ops_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/providers/cpu/rnn/deep_cpu_gru_op_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/providers/cpu/rnn/deep_cpu_lstm_op_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/providers/cpu/rnn/rnn_op_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/providers/cpu/sequence/concat_from_sequence_op_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/providers/cpu/sequence/sequence_ops_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/providers/cpu/tensor/cast_op_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/providers/cpu/tensor/compress_op.test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/providers/cpu/tensor/concat_op_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/providers/cpu/tensor/dynamic_quantize_linear_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/providers/cpu/tensor/expand_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/providers/cpu/tensor/eyelike_op_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/providers/cpu/tensor/gather_elements_op_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/providers/cpu/tensor/gather_nd_op_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/providers/cpu/tensor/gather_op_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/providers/cpu/tensor/identity_op_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/providers/cpu/tensor/isinf_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/providers/cpu/tensor/isnan_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/providers/cpu/tensor/nonzero_op_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/providers/cpu/tensor/onehot_op_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/providers/cpu/tensor/pad_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/providers/cpu/tensor/quantize_linear_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/providers/cpu/tensor/resize_op_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/providers/cpu/tensor/reverse_sequence_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/providers/cpu/tensor/scatter_nd_op_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/providers/cpu/tensor/scatter_op_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/providers/cpu/tensor/shape_op_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/providers/cpu/tensor/size_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/providers/cpu/tensor/slice_op.test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/providers/cpu/tensor/space_depth_ops_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/providers/cpu/tensor/split_op_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/providers/cpu/tensor/squeeze_op_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/providers/cpu/tensor/tensor_op_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/providers/cpu/tensor/tile_op_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/providers/cpu/tensor/transpose_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/providers/cpu/tensor/trilu_op_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/providers/cpu/tensor/trilu_shape_inference_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/providers/cpu/tensor/unique_op_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/providers/cpu/tensor/unsqueeze_op_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/providers/cpu/tensor/upsample_op_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/providers/cpu/tensor/where_op_test.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/providers/internal_testing/internal_testing_execution_provider.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/providers/internal_testing/internal_testing_tests.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/shared_lib/test_session_options.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/shared_lib/test_run_options.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/shared_lib/test_allocator.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/shared_lib/test_nontensor_types.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/shared_lib/test_model_loading.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/shared_lib/test_ort_format_models.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/shared_lib/utils.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/shared_lib/custom_op_utils.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/shared_lib/test_inference.cc.o CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/unittest_main/test_main.cc.o -o onnxruntime_test_all  libonnx_test_runner_common.a libonnxruntime_test_utils.a libonnxruntime_session.a libonnxruntime_optimizer.a libonnxruntime_providers.a libonnxruntime_util.a libonnxruntime_framework.a libonnxruntime_graph.a libonnxruntime_common.a libonnxruntime_mlas.a libonnxruntime_flatbuffers.a external/re2/libre2.a libonnx_test_data_proto.a lib/libgtest.a lib/libgmock.a external/onnx/libonnx.a external/onnx/libonnx_proto.a external/protobuf/cmake/libprotobuf-lite.a external/re2/libre2.a external/flatbuffers/libflatbuffers.a -latomic external/nsync/libnsync_cpp.a -ldl -lrt lib/libgtest.a -lpthread 
CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/providers/cpu/controlflow/loop_test.cc.o: In function `std::__atomic_futex_unsigned<2147483648u>::_M_store_notify_all(unsigned int, std::memory_order)':
/usr/arm-linux-gnueabihf/include/c++/7/bits/atomic_futex.h:208: undefined reference to `std::__atomic_futex_unsigned_base::_M_futex_notify_all(unsigned int*)'
/usr/arm-linux-gnueabihf/include/c++/7/bits/atomic_futex.h:208: undefined reference to `std::__atomic_futex_unsigned_base::_M_futex_notify_all(unsigned int*)'
/usr/arm-linux-gnueabihf/include/c++/7/bits/atomic_futex.h:208: undefined reference to `std::__atomic_futex_unsigned_base::_M_futex_notify_all(unsigned int*)'
CMakeFiles/onnxruntime_test_all.dir/onnxruntime/test/providers/cpu/controlflow/loop_test.cc.o: In function `std::__atomic_futex_unsigned<2147483648u>::_M_load_and_test_until(unsigned int, unsigned int, bool, std::memory_order, bool, std::chrono::duration<long long, std::ratio<1ll, 1ll> >, std::chrono::duration<long long, std::ratio<1ll, 1000000000ll> >)':
/usr/arm-linux-gnueabihf/include/c++/7/bits/atomic_futex.h:102: undefined reference to `std::__atomic_futex_unsigned_base::_M_futex_wait_until(unsigned int*, unsigned int, bool, std::chrono::duration<long long, std::ratio<1ll, 1ll> >, std::chrono::duration<long long, std::ratio<1ll, 1000000000ll> >)'
collect2: error: ld returned 1 exit status
CMakeFiles/onnxruntime_test_all.dir/build.make:3589: recipe for target 'onnxruntime_test_all' failed
make[2]: *** [onnxruntime_test_all] Error 1
make[2]: Leaving directory '/home/conan/onnxruntime'
CMakeFiles/Makefile2:1072: recipe for target 'CMakeFiles/onnxruntime_test_all.dir/all' failed
make[1]: *** [CMakeFiles/onnxruntime_test_all.dir/all] Error 2
make[1]: Leaving directory '/home/conan/onnxruntime'
Makefile:181: recipe for target 'all' failed
make: *** [all] Error 2

[snnn]

So atomic as already added to the link command. It could because the GCC version in your host os is bigger than the than used for cross-compiling(the ARM). I saw your host OS has GCC 7.5.0 but your ARM image is 7.2.0?

I don't quite understand whether you were doing cross-compiling or not. Did you run the build command directly in docker or in the host OS?

[drux007]

I don't quite understand whether you were doing cross-compiling or not. Did you run the build command directly in docker or in the host OS?

Sorry if it wasn't clear. I just added my host OS details while following the Issue template.

I run build command directly in docker container conanio/gcc7-armv7 (on my Ubuntu machine).

[drux007]

I saw your host OS has GCC 7.5.0 but your ARM image is 7.2.0?

So different gcc versions should not make any difference in this case, right?

[snnn]

Then you don't need to add "--path_to_protoc_exe" and the other stuffs in your build command. You were running in the build in an emulator(qemu). It is not what traditionally called cross-compiling. You may just build the build as usual as what typically on x86. like:

./build.sh  --parallel --config Release  --build_wheel

[drux007]

I am specifying CMAKE_C_COMPILER and CMAKE_CXX_COMPILER because I need to use hard float abi. That's why I am using arm-linux-gnueabihf-gcc-7 toolchain (instead of default arm-linux-gnueabi-gcc-7).
Please see #8384 for more details.

I also have to specify pre-compiled protoc because otherwise, I get the following error:

make[2]: external/protobuf/cmake/protoc-3.16.0.0: Command not found

E.g. if I run the command you suggested (without --build_wheel since I don't need it) and by specifying the hf toolchain:

/build.sh --cmake_extra_defines="CMAKE_C_COMPILER=/usr/bin/arm-linux-gnueabihf-gcc-7" --cmake_extra_defines="CMAKE_CXX_COMPILER=/usr/bin/arm-linux-gnueabihf-g++-7" --parallel --config Release

I get the following error:

[ 31%] Running C++ protocol buffer compiler on /home/conan/onnxruntime/build/Linux/Release/external/onnx/onnx/onnx-ml.proto
/lib/ld-linux-armhf.so.3: No such file or directory
external/onnx/CMakeFiles/gen_onnx_proto.dir/build.make:80: recipe for target 'external/onnx/onnx/onnx-ml.pb.cc' failed
make[2]: *** [external/onnx/onnx/onnx-ml.pb.cc] Error 255
CMakeFiles/Makefile2:2159: recipe for target 'external/onnx/CMakeFiles/gen_onnx_proto.dir/all' failed
make[1]: *** [external/onnx/CMakeFiles/gen_onnx_proto.dir/all] Error 2
Makefile:181: recipe for target 'all' failed
make: *** [all] Error 2
Traceback (most recent call last):
  File "/home/conan/onnxruntime/tools/ci_build/build.py", line 2152, in <module>
    sys.exit(main())
  File "/home/conan/onnxruntime/tools/ci_build/build.py", line 2076, in main
    build_targets(args, cmake_path, build_dir, configs, num_parallel_jobs, args.target)
  File "/home/conan/onnxruntime/tools/ci_build/build.py", line 1067, in build_targets
    run_subprocess(cmd_args, env=env)
  File "/home/conan/onnxruntime/tools/ci_build/build.py", line 581, in run_subprocess
    return run(*args, cwd=cwd, capture_stdout=capture_stdout, shell=shell, env=my_env)
  File "/home/conan/onnxruntime/tools/python/util/run.py", line 44, in run
    env=env, shell=shell)
  File "/opt/pyenv/versions/3.7.5/lib/python3.7/subprocess.py", line 512, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['/usr/bin/cmake', '--build', '/home/conan/onnxruntime/build/Linux/Release', '--config', 'Release', '--', '-j8']' returned non-zero exit status 2.

[snnn]

Ok, I got it. conanio/gcc7-armv7 is a x86_64 image, not arm.

[snnn]

I don't know much about conanio/gcc7-armv7. I'm afraid I can't provide much help on that. It is a x86_64 image, so if you don't have a prebuilt protoc and build it from our source tree, you will get one in ARM format. You can't run an ARM binary in an x64_64 docker image. That's why you got the "make[2]: external/protobuf/cmake/protoc-3.16.0.0: Command not found" error.

But if you want to build onnx runtime in an x64 environment, I suggest you doing it directly in your host OS. Because my crosss-compiling instructions rely on docker. You can't run docker inside docker. And things will get very complicated. It is beyond of my knowledge. And when you do cross-compiling, like

cmake -DCMAKE_CXX_COMPILER=/usr/bin/arm-linux-gnueabihf-g++-7 -DCMAKE_C_COMPILER=/usr/bin/arm-linux-gnueabihf-gcc-7 -DONNX_CUSTOM_PROTOC_EXECUTABLE=/home/conan/protoc-3.16.0-linux-x86_64/bin/protoc ./cmake

You must setup a toolchain file for it. For example:

 SET(CMAKE_SYSTEM_NAME Linux)
 SET(CMAKE_SYSTEM_VERSION 1)
 SET(CMAKE_C_COMPILER aarch64-linux-gnu-gcc)
 SET(CMAKE_CXX_COMPILER aarch64-linux-gnu-g++)
 SET(CMAKE_FIND_ROOT_PATH_MODE_PROGRAM NEVER)
 SET(CMAKE_FIND_ROOT_PATH_MODE_LIBRARY ONLY)
 SET(CMAKE_FIND_ROOT_PATH_MODE_INCLUDE ONLY)
 SET(CMAKE_FIND_ROOT_PATH_MODE_PACKAGE ONLY)

In the third and forth line, you can use abs paths. But, you will need all these settings. Currently you only set two.

Or, you may do it in a real ARM docker image(like https://hub.docker.com/r/arm32v7/ubuntu/). You need to figure out which OS your ARM device uses. Then you must use the same OS and same version. It will be much easier to setup but also much slower. If you don't need to develop onnxruntime and run the build multiple times everyday, I feel the build time is acceptable.

[snnn]

And the link error you reported is more like a compiler bug.

See:
richfelker/musl-cross-make#79
https://stackoverflow.com/questions/46999056/crosscompile-undefined-reference-to-std-atomic-futex-unsigned-base-m-futex

Both of the threads suggest using a newer version of GCC. Therefore I will close this issue now.