[Build] /usr/bin/ld: failed to set dynamic section sizes: memory exhausted #15337

goldyfruit · 2023-04-03T17:25:58Z

Describe the issue

Unable to build onnxruntime on Debian Bookworm container for armv7.

Urgency

No response

Target platform

armv7

Build script

Dockerfile-debian.arm32v7

FROM arm32v7/debian:bookworm
ADD . /code

RUN /code/dockerfiles/scripts/install_debian_armv7.sh && cd /code && ./build.sh --parallel --build_wheel --skip_test --allow_running_as_root --skip_submodule_sync --update --build --cmake_extra_defines ONNXRUNTIME_VERSION=$(cat ./VERSION_NUMBER)

install_debian_armv7.sh

apt-get update
apt-get install -y python3-pip python3-wheel python3-dev python3-numpy python3-packaging git build-essential gcc g++ git cmake protobuf-compiler libprotoc-dev python3-setuptools

cd onnxruntime
docker build -t onnxruntime-source -f dockerfiles/Dockerfile-debian.arm32v7 .
```

### Error / output

```#0 7614.1       |   ~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
#0 7614.1 /usr/bin/ld: warning: QgemmU8X8KernelNeon.S.o: missing .note.GNU-stack section implies executable stack
#0 7614.1 /usr/bin/ld: NOTE: This behaviour is deprecated and will be removed in a future version of the linker
#0 7614.1 /usr/bin/ld: failed to set dynamic section sizes: memory exhausted
#0 7614.1 collect2: error: ld returned 1 exit status
#0 7614.1 gmake[2]: *** [CMakeFiles/onnxruntime_test_all.dir/build.make:4768: onnxruntime_test_all] Error 1
#0 7614.1 gmake[1]: *** [CMakeFiles/Makefile2:2130: CMakeFiles/onnxruntime_test_all.dir/all] Error 2
#0 7614.1 gmake: *** [Makefile:166: all] Error 2
#0 7614.1 Traceback (most recent call last):
#0 7614.1   File "/code/tools/ci_build/build.py", line 2797, in <module>
#0 7614.1     sys.exit(main())
#0 7614.1              ^^^^^^
#0 7614.1   File "/code/tools/ci_build/build.py", line 2693, in main
#0 7614.1     build_targets(args, cmake_path, build_dir, configs, num_parallel_jobs, args.target)
#0 7614.1   File "/code/tools/ci_build/build.py", line 1444, in build_targets
#0 7614.1     run_subprocess(cmd_args, env=env)
#0 7614.1   File "/code/tools/ci_build/build.py", line 785, in run_subprocess
#0 7614.1     return run(*args, cwd=cwd, capture_stdout=capture_stdout, shell=shell, env=my_env)
#0 7614.1            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
#0 7614.1   File "/code/tools/python/util/run.py", line 49, in run
#0 7614.1     completed_process = subprocess.run(
#0 7614.1                         ^^^^^^^^^^^^^^^
#0 7614.1   File "/usr/lib/python3.11/subprocess.py", line 571, in run
#0 7614.1     raise CalledProcessError(retcode, process.args,
#0 7614.1 subprocess.CalledProcessError: Command '['/usr/bin/cmake', '--build', '/code/build/Linux/Debug', '--config', 'Debug', '--', '-j32']' returned non-zero exit status 2.
```

### Visual Studio Version

_No response_

### GCC / Compiler Version

12.2.0

The text was updated successfully, but these errors were encountered:

baijumeswani · 2023-04-04T16:24:53Z

From the linker error message, it seems like the machine doesn't have enough ram to link the code.

I am not sure if there is a solution to this problem other than increasing machine ram. Maybe @snnn has some helpful insight for this?

goldyfruit · 2023-04-04T16:54:36Z

@baijumeswani thanks for your answer, I got 256GB of RAM total on the server and 182GB are free. I was able as well to build the aarch64 version on the same server.

Maybe it's related to a 4GB limit, don't really know.

baijumeswani · 2023-04-04T21:00:11Z

@goldyfruit thanks for the info. I should not have assumed. :)

I'm sorry but I have no good reason why you're seeing the linker error. I'll try to consult with other engineers internally, but not sure if I will have any luck.

If you do find the answer to the problem, please post it here so it can benefit others who might find themselves in a similar boat.

goldyfruit · 2023-04-04T21:48:31Z

@goldyfruit thanks for the info. I should not have assumed. :)

You couldn't 👍

I'm sorry but I have no good reason why you're seeing the linker error. I'll try to consult with other engineers internally, but not sure if I will have any luck.

I hope you will have some update.

If you do find the answer to the problem, please post it here so it can benefit others who might find themselves in a similar boat.

Sure.

satyajandhyala · 2023-05-05T22:05:59Z

@goldyfruit Did you try removing "--parallel" build.sh command-line option?

goldyfruit · 2023-05-05T22:17:48Z

@goldyfruit Did you try removing "--parallel" build.sh command-line option?

No I don't, let me try without it!

goldyfruit · 2023-05-06T11:33:42Z

Still running... 13 hours!

goldyfruit · 2023-05-07T12:39:04Z

Same error after 27 hours of build.

#0 99401.8 /usr/bin/ld: warning: QgemmU8X8KernelNeon.S.o: missing .note.GNU-stack section implies executable stack
#0 99401.8 /usr/bin/ld: NOTE: This behaviour is deprecated and will be removed in a future version of the linker
#0 99401.8 /usr/bin/ld: failed to set dynamic section sizes: memory exhausted
#0 99401.8 collect2: error: ld returned 1 exit status
#0 99401.8 gmake[2]: *** [CMakeFiles/onnxruntime_test_all.dir/build.make:4768: onnxruntime_test_all] Error 1
#0 99401.8 gmake[1]: *** [CMakeFiles/Makefile2:2130: CMakeFiles/onnxruntime_test_all.dir/all] Error 2
#0 99401.8 gmake: *** [Makefile:166: all] Error 2
#0 99401.8 Traceback (most recent call last):
#0 99401.8   File "/code/tools/ci_build/build.py", line 2773, in <module>
#0 99401.8     sys.exit(main())
#0 99401.8              ^^^^^^
#0 99401.8   File "/code/tools/ci_build/build.py", line 2669, in main
#0 99401.8     build_targets(args, cmake_path, build_dir, configs, num_parallel_jobs, args.target)
#0 99401.8   File "/code/tools/ci_build/build.py", line 1447, in build_targets
#0 99401.8     run_subprocess(cmd_args, env=env)
#0 99401.8   File "/code/tools/ci_build/build.py", line 787, in run_subprocess
#0 99401.8     return run(*args, cwd=cwd, capture_stdout=capture_stdout, shell=shell, env=my_env)
#0 99401.8            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
#0 99401.8   File "/code/tools/python/util/run.py", line 49, in run
#0 99401.8     completed_process = subprocess.run(
#0 99401.8                         ^^^^^^^^^^^^^^^
#0 99401.8   File "/usr/lib/python3.11/subprocess.py", line 571, in run
#0 99401.8     raise CalledProcessError(retcode, process.args,
#0 99401.8 subprocess.CalledProcessError: Command '['/usr/bin/cmake', '--build', '/code/build/Linux/Debug', '--config', 'Debug']' returned non-zero exit status 2.

satyajandhyala · 2023-05-08T22:13:31Z

Can you try building a Release version instead of a Debug version? I am hoping that Release build might succeed. You can specify the release build by adding " --config Release" option to the build.sh command.

satyajandhyala · 2023-05-11T16:50:18Z

@goldyfruit Did you try building Release build?

goldyfruit · 2023-05-11T17:01:55Z

@goldyfruit Did you try building Release build?

Build in progress (thanks for the reminder 👍 )

goldyfruit · 2023-05-12T00:25:21Z

Did a git pull and now it requires Cmake 3.26...

#0 3828.7 CMake Error at CMakeLists.txt:11 (cmake_minimum_required):
#0 3828.7   CMake 3.26 or higher is required.  You are running version 3.25.1
#0 3828.7 
#0 3828.7 
#0 3828.7 -- Configuring incomplete, errors occurred!
#0 3828.7 Traceback (most recent call last):
#0 3828.7   File "/code/tools/ci_build/build.py", line 2601, in <module>
#0 3828.7     sys.exit(main())
#0 3828.7              ^^^^^^
#0 3828.7   File "/code/tools/ci_build/build.py", line 2462, in main
#0 3828.7     generate_build_tree(
#0 3828.7   File "/code/tools/ci_build/build.py", line 1371, in generate_build_tree
#0 3828.7     run_subprocess(
#0 3828.7   File "/code/tools/ci_build/build.py", line 780, in run_subprocess
#0 3828.7     return run(*args, cwd=cwd, capture_stdout=capture_stdout, shell=shell, env=my_env)
#0 3828.7            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
#0 3828.7   File "/code/tools/python/util/run.py", line 49, in run
#0 3828.7     completed_process = subprocess.run(
#0 3828.7                         ^^^^^^^^^^^^^^^
#0 3828.7   File "/usr/lib/python3.11/subprocess.py", line 571, in run
#0 3828.7     raise CalledProcessError(retcode, process.args,
#0 3828.7 subprocess.CalledProcessError: Command '['/usr/bin/cmake', '/code/cmake', '-Donnxruntime_RUN_ONNX_TESTS=OFF', '-Donnxruntime_GENERATE_TEST_REPORTS=ON', '-DPython_EXECUTABLE=/usr/bin/python3', '-DPYTHON_EXECUTABLE=/usr/bin/python3', '-Donnxruntime_USE_MIMALLOC=OFF', '-Donnxruntime_ENABLE_PYTHON=ON', '-Donnxruntime_BUILD_CSHARP=OFF', '-Donnxruntime_BUILD_JAVA=OFF', '-Donnxruntime_BUILD_NODEJS=OFF', '-Donnxruntime_BUILD_OBJC=OFF', '-Donnxruntime_BUILD_SHARED_LIB=OFF', '-Donnxruntime_BUILD_APPLE_FRAMEWORK=OFF', '-Donnxruntime_USE_DNNL=OFF', '-Donnxruntime_USE_NNAPI_BUILTIN=OFF', '-Donnxruntime_USE_RKNPU=OFF', '-Donnxruntime_USE_LLVM=OFF', '-Donnxruntime_ENABLE_MICROSOFT_INTERNAL=OFF', '-Donnxruntime_USE_VITISAI=OFF', '-Donnxruntime_USE_TENSORRT=OFF', '-Donnxruntime_USE_TENSORRT_BUILTIN_PARSER=ON', '-Donnxruntime_USE_TVM=OFF', '-Donnxruntime_TVM_CUDA_RUNTIME=OFF', '-Donnxruntime_TVM_USE_HASH=OFF', '-Donnxruntime_USE_MIGRAPHX=OFF', '-Donnxruntime_DISABLE_CONTRIB_OPS=OFF', '-Donnxruntime_DISABLE_ML_OPS=OFF', '-Donnxruntime_DISABLE_RTTI=OFF', '-Donnxruntime_DISABLE_EXCEPTIONS=OFF', '-Donnxruntime_MINIMAL_BUILD=OFF', '-Donnxruntime_EXTENDED_MINIMAL_BUILD=OFF', '-Donnxruntime_MINIMAL_BUILD_CUSTOM_OPS=OFF', '-Donnxruntime_REDUCED_OPS_BUILD=OFF', '-Donnxruntime_ENABLE_LANGUAGE_INTEROP_OPS=OFF', '-Donnxruntime_USE_DML=OFF', '-Donnxruntime_USE_WINML=OFF', '-Donnxruntime_BUILD_MS_EXPERIMENTAL_OPS=OFF', '-Donnxruntime_USE_TELEMETRY=OFF', '-Donnxruntime_ENABLE_LTO=OFF', '-Donnxruntime_USE_ACL=OFF', '-Donnxruntime_USE_ACL_1902=OFF', '-Donnxruntime_USE_ACL_1905=OFF', '-Donnxruntime_USE_ACL_1908=OFF', '-Donnxruntime_USE_ACL_2002=OFF', '-Donnxruntime_USE_ARMNN=OFF', '-Donnxruntime_ARMNN_RELU_USE_CPU=ON', '-Donnxruntime_ARMNN_BN_USE_CPU=ON', '-Donnxruntime_USE_JSEP=OFF', '-Donnxruntime_ENABLE_NVTX_PROFILE=OFF', '-Donnxruntime_ENABLE_TRAINING=OFF', '-Donnxruntime_ENABLE_TRAINING_OPS=OFF', '-Donnxruntime_ENABLE_TRAINING_APIS=OFF', '-Donnxruntime_ENABLE_CPU_FP16_OPS=OFF', '-Donnxruntime_USE_NCCL=OFF', '-Donnxruntime_BUILD_BENCHMARKS=OFF', '-Donnxruntime_USE_ROCM=OFF', '-DOnnxruntime_GCOV_COVERAGE=OFF', '-Donnxruntime_USE_MPI=OFF', '-Donnxruntime_ENABLE_MEMORY_PROFILE=OFF', '-Donnxruntime_ENABLE_CUDA_LINE_NUMBER_INFO=OFF', '-Donnxruntime_BUILD_WEBASSEMBLY=OFF', '-Donnxruntime_BUILD_WEBASSEMBLY_STATIC_LIB=OFF', '-Donnxruntime_ENABLE_WEBASSEMBLY_EXCEPTION_CATCHING=ON', '-Donnxruntime_ENABLE_WEBASSEMBLY_API_EXCEPTION_CATCHING=OFF', '-Donnxruntime_ENABLE_WEBASSEMBLY_EXCEPTION_THROWING=ON', '-Donnxruntime_WEBASSEMBLY_RUN_TESTS_IN_BROWSER=OFF', '-Donnxruntime_ENABLE_WEBASSEMBLY_THREADS=OFF', '-Donnxruntime_ENABLE_WEBASSEMBLY_DEBUG_INFO=OFF', '-Donnxruntime_ENABLE_WEBASSEMBLY_PROFILING=OFF', '-Donnxruntime_ENABLE_LAZY_TENSOR=OFF', '-Donnxruntime_ENABLE_EXTERNAL_CUSTOM_OP_SCHEMAS=OFF', '-Donnxruntime_ENABLE_CUDA_PROFILING=OFF', '-Donnxruntime_ENABLE_ROCM_PROFILING=OFF', '-Donnxruntime_USE_XNNPACK=OFF', '-Donnxruntime_USE_WEBNN=OFF', '-Donnxruntime_USE_CANN=OFF', '-DONNXRUNTIME_VERSION=1.16.0', '-DCMAKE_TLS_VERIFY=ON', '-DFETCHCONTENT_QUIET=OFF', '-Donnxruntime_ENABLE_MEMLEAK_CHECKER=OFF', '-DCMAKE_BUILD_TYPE=Release', '-DCMAKE_PREFIX_PATH=/code/build/Linux/Release/installed']' returned non-zero exit status 1.
------

I give up for now, thanks for helping.

satyajandhyala · 2023-05-12T00:42:37Z

When you get back to this, you can install/upgrade cmake using "pip install cmake"

goldyfruit added the build build issues; typically submitted using template label Apr 3, 2023

goldyfruit closed this as completed May 12, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Build] /usr/bin/ld: failed to set dynamic section sizes: memory exhausted #15337

[Build] /usr/bin/ld: failed to set dynamic section sizes: memory exhausted #15337

goldyfruit commented Apr 3, 2023

baijumeswani commented Apr 4, 2023

goldyfruit commented Apr 4, 2023 •

edited

baijumeswani commented Apr 4, 2023

goldyfruit commented Apr 4, 2023

satyajandhyala commented May 5, 2023

goldyfruit commented May 5, 2023

goldyfruit commented May 6, 2023

goldyfruit commented May 7, 2023

satyajandhyala commented May 8, 2023 •

edited

satyajandhyala commented May 11, 2023

goldyfruit commented May 11, 2023

goldyfruit commented May 12, 2023 •

edited

satyajandhyala commented May 12, 2023

[Build] /usr/bin/ld: failed to set dynamic section sizes: memory exhausted #15337

[Build] /usr/bin/ld: failed to set dynamic section sizes: memory exhausted #15337

Comments

goldyfruit commented Apr 3, 2023

Describe the issue

Urgency

Target platform

Build script

baijumeswani commented Apr 4, 2023

goldyfruit commented Apr 4, 2023 • edited

baijumeswani commented Apr 4, 2023

goldyfruit commented Apr 4, 2023

satyajandhyala commented May 5, 2023

goldyfruit commented May 5, 2023

goldyfruit commented May 6, 2023

goldyfruit commented May 7, 2023

satyajandhyala commented May 8, 2023 • edited

satyajandhyala commented May 11, 2023

goldyfruit commented May 11, 2023

goldyfruit commented May 12, 2023 • edited

satyajandhyala commented May 12, 2023

goldyfruit commented Apr 4, 2023 •

edited

satyajandhyala commented May 8, 2023 •

edited

goldyfruit commented May 12, 2023 •

edited