Description
Since flang-rt builds were added to the flang-aarch64-libcxx bot it has been failing with:
FAILED: flang-rt/unittests/Runtime/RuntimeTests
: && /home/tcwg-buildbot/worker/flang-aarch64-libcxx/build/./bin/clang++ --target=aarch64-unknown-linux-gnu -fPIC -fno-semantic-interposition -fvisibility-inlines-hidden -Werror=date-time -Werror=unguarded-availability-new -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -Wimplicit-fallthrough -Wcovered-switch-default -Wno-noexcept-type -Wnon-virtual-dtor -Wdelete-non-virtual-dtor -Wsuggest-override -Wstring-conversion -Wmisleading-indentation -Wctad-maybe-unsupported -fdiagnostics-color -ffunction-sections -fdata-sections -O3 -DNDEBUG -Wl,--gc-sections flang-rt/unittests/Runtime/CMakeFiles/RuntimeTests.dir/AccessTest.cpp.o flang-rt/unittests/Runtime/CMakeFiles/RuntimeTests.dir/Allocatable.cpp.o flang-rt/unittests/Runtime/CMakeFiles/RuntimeTests.dir/ArrayConstructor.cpp.o flang-rt/unittests/Runtime/CMakeFiles/RuntimeTests.dir/BufferTest.cpp.o flang-rt/unittests/Runtime/CMakeFiles/RuntimeTests.dir/CharacterTest.cpp.o flang-rt/unittests/Runtime/CMakeFiles/RuntimeTests.dir/CommandTest.cpp.o flang-rt/unittests/Runtime/CMakeFiles/RuntimeTests.dir/Complex.cpp.o flang-rt/unittests/Runtime/CMakeFiles/RuntimeTests.dir/CrashHandlerFixture.cpp.o flang-rt/unittests/Runtime/CMakeFiles/RuntimeTests.dir/Derived.cpp.o flang-rt/unittests/Runtime/CMakeFiles/RuntimeTests.dir/ExternalIOTest.cpp.o flang-rt/unittests/Runtime/CMakeFiles/RuntimeTests.dir/Format.cpp.o flang-rt/unittests/Runtime/CMakeFiles/RuntimeTests.dir/Inquiry.cpp.o flang-rt/unittests/Runtime/CMakeFiles/RuntimeTests.dir/ListInputTest.cpp.o flang-rt/unittests/Runtime/CMakeFiles/RuntimeTests.dir/LogicalFormatTest.cpp.o flang-rt/unittests/Runtime/CMakeFiles/RuntimeTests.dir/Matmul.cpp.o flang-rt/unittests/Runtime/CMakeFiles/RuntimeTests.dir/MatmulTranspose.cpp.o flang-rt/unittests/Runtime/CMakeFiles/RuntimeTests.dir/MiscIntrinsic.cpp.o flang-rt/unittests/Runtime/CMakeFiles/RuntimeTests.dir/Namelist.cpp.o flang-rt/unittests/Runtime/CMakeFiles/RuntimeTests.dir/Numeric.cpp.o flang-rt/unittests/Runtime/CMakeFiles/RuntimeTests.dir/NumericalFormatTest.cpp.o flang-rt/unittests/Runtime/CMakeFiles/RuntimeTests.dir/Pointer.cpp.o flang-rt/unittests/Runtime/CMakeFiles/RuntimeTests.dir/Ragged.cpp.o flang-rt/unittests/Runtime/CMakeFiles/RuntimeTests.dir/Random.cpp.o flang-rt/unittests/Runtime/CMakeFiles/RuntimeTests.dir/Reduction.cpp.o flang-rt/unittests/Runtime/CMakeFiles/RuntimeTests.dir/RuntimeCrashTest.cpp.o flang-rt/unittests/Runtime/CMakeFiles/RuntimeTests.dir/Stop.cpp.o flang-rt/unittests/Runtime/CMakeFiles/RuntimeTests.dir/Support.cpp.o flang-rt/unittests/Runtime/CMakeFiles/RuntimeTests.dir/Time.cpp.o flang-rt/unittests/Runtime/CMakeFiles/RuntimeTests.dir/TemporaryStack.cpp.o flang-rt/unittests/Runtime/CMakeFiles/RuntimeTests.dir/Transformational.cpp.o -o flang-rt/unittests/Runtime/RuntimeTests -Wl,-rpath,/home/tcwg-buildbot/worker/flang-aarch64-libcxx/build/lib /home/tcwg-buildbot/worker/flang-aarch64-libcxx/build/lib/libllvm_gtest_main.a /home/tcwg-buildbot/worker/flang-aarch64-libcxx/build/lib/libllvm_gtest.a /home/tcwg-buildbot/worker/flang-aarch64-libcxx/build/lib/clang/21/lib/aarch64-unknown-linux-gnu/libflang_rt.runtime.a /home/tcwg-buildbot/worker/flang-aarch64-libcxx/build/lib/libLLVMSupport.so.21.0git -Wl,-rpath-link,/home/tcwg-buildbot/worker/flang-aarch64-libcxx/build/lib && :
/usr/bin/ld: flang-rt/unittests/Runtime/CMakeFiles/RuntimeTests.dir/AccessTest.cpp.o: in function `createTemporaryFile[abi:cxx11](char const*, (anonymous namespace)::AccessType const&)':
AccessTest.cpp:(.text._ZL19createTemporaryFileB5cxx11PKcRKN12_GLOBAL__N_110AccessTypeE+0x194): undefined reference to `llvm::Twine::str[abi:cxx11]() const'
clang++: error: linker command failed with exit code 1 (use -v to see invocation)
ninja: build stopped: subcommand failed.
The theory is that this is because the host compiler includes libc++ and uses that, but the stage 1 compiler does not. So it's mixing libc++ stuff built by the host compiler with libstdc++ things it has built as part of the runtimes.
#134990 aims to fix this by passing on the LLVM_ENABLE_LIBCXX setting to the runtimes build.
The theory is sound but the bot still failed:
FAILED: flang-rt/lib/runtime/CMakeFiles/flang_rt.runtime.static.dir/allocator-registry.cpp.o
/home/tcwg-buildbot/worker/flang-aarch64-libcxx/build/./bin/clang++ --target=aarch64-unknown-linux-gnu -D_DEBUG -D_GLIBCXX_ASSERTIONS -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -I/home/tcwg-buildbot/worker/flang-aarch64-libcxx/llvm-project/flang-rt/include -I/home/tcwg-buildbot/worker/flang-aarch64-libcxx/llvm-project/flang-rt/../flang/include -I/home/tcwg-buildbot/worker/flang-aarch64-libcxx/build/runtimes/runtimes-bins/flang-rt -stdlib=libc++ -fvisibility-inlines-hidden -Werror=date-time -Werror=unguarded-availability-new -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -Wimplicit-fallthrough -Wcovered-switch-default -Wno-noexcept-type -Wnon-virtual-dtor -Wdelete-non-virtual-dtor -Wsuggest-override -Wstring-conversion -fdiagnostics-color -ffunction-sections -fdata-sections -O3 -DNDEBUG -std=gnu++17 -UNDEBUG -fno-lto -fno-exceptions -fno-rtti -funwind-tables -fno-asynchronous-unwind-tables -U_GLIBCXX_ASSERTIONS -U_LIBCPP_ENABLE_ASSERTIONS -MD -MT flang-rt/lib/runtime/CMakeFiles/flang_rt.runtime.static.dir/allocator-registry.cpp.o -MF flang-rt/lib/runtime/CMakeFiles/flang_rt.runtime.static.dir/allocator-registry.cpp.o.d -o flang-rt/lib/runtime/CMakeFiles/flang_rt.runtime.static.dir/allocator-registry.cpp.o -c /home/tcwg-buildbot/worker/flang-aarch64-libcxx/llvm-project/flang-rt/lib/runtime/allocator-registry.cpp
In file included from /home/tcwg-buildbot/worker/flang-aarch64-libcxx/llvm-project/flang-rt/lib/runtime/allocator-registry.cpp:9:
/home/tcwg-buildbot/worker/flang-aarch64-libcxx/llvm-project/flang-rt/include/flang-rt/runtime/allocator-registry.h:14:10: fatal error: 'cstdint' file not found
14 | #include <cstdint>
| ^~~~~~~~~
This makes sense given that libcxx is not in LLVM_ENABLE_RUNTIMES, and the stage 1 compiler has no way to know the other host compiler exists in this particular bot's setup. Which is...
- An llvm release package is copied into /usr/local/ and "cc" and "cxx" are setup to point to clang and clang++ in that package.
- The library directory of that package is added to ld.so.conf. This means that libc++ is not installed system wide, this is important.
So what can we do to fix this?
I think the PR itself is fine, but the bot needs to be updated to work with it.
We need to:
- Not test flang and flang-rt using mismatched versions of libc++ and/or libstdc++.
- Not hard code compiler paths into llvm-zorg, which would be a major pain when we come to update the host compiler.
Some routes we could take...
Pass the include and library paths to the stage 1 compiler
cmake -DLLVM_TARGETS_TO_BUILD=AArch64 -DLLVM_INSTALL_UTILS=ON -DCMAKE_CXX_STANDARD=17 \
-DLLVM_ENABLE_WERROR=OFF -DFLANG_ENABLE_WERROR=ON -DBUILD_SHARED_LIBS=ON \
-DLLVM_ENABLE_ASSERTIONS=ON -DLLVM_ENABLE_LIBCXX=On -DCMAKE_BUILD_TYPE=Release \
-DLLVM_RUNTIME_TARGETS=aarch64-unknown-linux-gnu \
-DRUNTIMES_aarch64-unknown-linux-gnu_CMAKE_CXX_FLAGS="-I /usr/local/clang+llvm-19.1.7-aarch64-linux-gnu/include/c++/v1/ -I /usr/local/clang+llvm-19.1.7-aarch64-linux-gnu/include/aarch64-unknown-linux-gnu/c++/v1/ -L/usr/local/clang+llvm-19.1.7-aarch64-linux-gnu/lib/aarch64-unknown-linux-gnu/" \
'-DLLVM_ENABLE_PROJECTS=llvm;clang;mlir;flang' -DLLVM_ENABLE_RUNTIMES=flang-rt \
'-DLLVM_LIT_ARGS=-v -vv' -GNinja ../llvm-project/llvm && \
ninja && \
ninja check-flang check-flang-rt-aarch64-unknown-linux-gnu
Note that:
- We must specify LLVM_RUNTIME_TARGETS to be able to pass custom options into the runtimes build.
- Doing so changes the check target name for flang-rt.
This works but we would have to hardcode those paths, or put stable symlinks in our docker image and point to those. Which is not ideal as it's one more step to reproduce outside of the container for what is, in theory, a pretty simple build.
Build the runtime with the host compiler instead of stage 1 compiler
We have to build clang to build flang, but we don't have to use the just built clang:
cmake -DLLVM_TARGETS_TO_BUILD=AArch64 -DLLVM_INSTALL_UTILS=ON -DCMAKE_CXX_STANDARD=17 \
-DLLVM_ENABLE_WERROR=OFF -DFLANG_ENABLE_WERROR=ON -DBUILD_SHARED_LIBS=ON \
-DLLVM_ENABLE_ASSERTIONS=ON -DLLVM_ENABLE_LIBCXX=On -DCMAKE_BUILD_TYPE=Release \
-DLLVM_RUNTIME_TARGETS=aarch64-unknown-linux-gnu \
-DRUNTIMES_aarch64-unknown-linux-gnu_CMAKE_CXX_COMPILER="c++" \
-DRUNTIMES_aarch64-unknown-linux-gnu_CMAKE_C_COMPILER="cc" \
'-DLLVM_ENABLE_PROJECTS=llvm;clang;mlir;flang' -DLLVM_ENABLE_RUNTIMES=flang-rt \
-GNinja ../llvm-project/llvm && \
ninja && \
ninja check-flang-rt check-flang-rt-aarch64-unknown-linux-gnu
This is basically a standalone build of flang-rt (where you point cmake at flang-rt itself, not llvm) but with more steps. It does work but I'm not sure we're supposed to be doing it this way.
With the way our container is setup, "cc" will be clang and "cxx" clang++. So I'd be fine putting those into llvm-zorg as those names are stable.
Change the builder type and build flang-rt standalone using host compiler
Unified tree builder doesn't seem like it would accept an extra step easily, but flang-aarch64-out-of-tree uses out of tree builder which might. We end up with a bot that is unusual in 2 ways (out of tree and libc++) not one, but it's easier to fit into llvm-zorg.
Or we look at making it an annotated builder where there's total freedom. It's a bit much for a theoretically simple change though.
Add libcxx to runtimes (which does not work)
This one does not work, at least for our goals.
cmake -DLLVM_TARGETS_TO_BUILD=AArch64 -DLLVM_INSTALL_UTILS=ON -DCMAKE_CXX_STANDARD=17 \
-DLLVM_ENABLE_WERROR=OFF -DFLANG_ENABLE_WERROR=ON -DBUILD_SHARED_LIBS=ON \
-DLLVM_ENABLE_ASSERTIONS=ON -DLLVM_ENABLE_LIBCXX=On -DCMAKE_BUILD_TYPE=Release \
'-DLLVM_ENABLE_PROJECTS=llvm;clang;mlir;flang' -DLLVM_ENABLE_RUNTIMES="libcxx;libcxxabi;libunwind;flang-rt" \
-GNinja ../llvm-project/llvm && \
ninja && \
ninja check-flang check-flang-rt
This does give stage 1 clang a libcxx to use but the tests end up compiling against a newer libcxx than they would find when trying to run:
/home/tcwg-buildbot/build-llvm/runtimes/runtimes-bins/flang-rt/unittests/Runtime/./RuntimeTests:
symbol lookup error: /home/tcwg-buildbot/build-llvm/runtimes/runtimes-bins/flang-rt/unittests/Runtime/./RuntimeTests:
undefined symbol: _ZNSt3__113__hash_memoryEPKvm
You can fix that with LD_LIBRARY_PATH, but doing that in a bot config is a maintanence problem.
$ LD_LIBRARY_PATH=./runtimes/runtimes-bins/libcxxabi/test-suite-install/lib/aarch64-unknown-linux-gnu /home/tcwg-buildbot/build-llvm/runtimes/runtimes-bins/flang-rt/unittests/Runtime/./RuntimeTests
Even if we could get the tests to run this way, we now have tests using 1 version of libcxx, and flang using another. If we actually find a real bug, that's one more dimension to deal with.
So I do not think we should pursue this.
Recommendation
I think we either:
- Tell the runtimes build to use the host compiler, assuming this is not massively against the spirit of the runtimes build. We could also do this as a short term fix, even if it is not a great idea.
- Change the builder type and add a step building a standalone flang-rt using the host compiler.
Activity
DavidSpickett commentedon Apr 11, 2025
I'll be at Euro LLVM so I'm handing this off to a colleague to finish up.
Notes for them:
--entrypoint bash
option to skip starting the buildbot agent.run.sh
would have. That is the ld.so.conf file and the cc/cxx setup.luporl commentedon Apr 16, 2025
Thanks for detailed information @DavidSpickett.
I was able to reproduce this issue as well, also reaching the conclusion that the problem is mixing stuff built with libc++ with things built with libstdc++. Below I expand a bit more about the cause.
LLVM_ENABLE_LIBCXX=On
makes LLVM be built with-stdlib=libc++
.llvm::Twine::str() const
, defined in Twine.cpp.o, part of LLVM libraries, is built with this flag.But
AccessTest.cpp.o
, from flang-rt tests, that referencesllvm::Twine::str
, is built without-stdlib=libc++
, thus using libstdc++.As a result, when linking, the linker searches for
llvm::Twine::str[abi:cxx11]() const
instead ofllvm::Twine::str() const
, which is not found.Analyzing the possible solutions listed above, it seems to me that building the runtime with the host compiler instead of stage 1 compiler is the best one. It's relatively simple, doesn't require many changes, and, as the only runtime that flang-aarch64-libcxx builts is flang-rt, that doesn't have a strong dependency on the version of the C++ compiler and libraries used, it shouldn't be a problem.
I will prepare a PR with this change.
Fix flang-aarch64-libcxx builder
DavidSpickett commentedon Jun 11, 2025
We had a lot of things going on in the last couple of months, just getting back to this...
This confused me for a minute but you're right. We enable these things:
libcxx is already present in the host compiler's install.
So you're saying we can:
That sounds good to me. This will be the easiest for folks to understand and reproduce I think.
And now I see you made a PR already, off to review that...
Fix flang-aarch64-libcxx builder
Fix flang-aarch64-libcxx builder (#432)
1 remaining item