BOLT doesn't build with -DLLVM_LINK_LLVM_DYLIB on Debian bullseye

[sylvestre]

FAILED: bin/llvm-bolt-heatmap 
: && "/home/sylvestre/dev/debian/pkg-llvm/llvm-toolchain/branches/llvm-toolchain-15-15.0.0~+rc3/build-llvm/./bin/clang++" -fstack-protector-strong -Wformat -Werror=format-security -Wno-unused-command-line-argument -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -fno-semantic-interposition -fvisibility-inlines-hidden -Werror=date-time -Werror=unguarded-availability-new -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -pedantic -Wno-long-long -Wc++98-compat-extra-semi -Wimplicit-fallthrough -Wcovered-switch-default -Wno-noexcept-type -Wnon-virtual-dtor -Wdelete-non-virtual-dtor -Wsuggest-override -Wno-comment -Wstring-conversion -Wmisleading-indentation -fdiagnostics-color -ffunction-sections -fdata-sections -ffile-prefix-map=/home/sylvestre/dev/debian/pkg-llvm/llvm-toolchain/branches/llvm-toolchain-15-15.0.0~+rc3/build-llvm/tools/clang/stage2-bins=build-llvm/tools/clang/stage2-bins -ffile-prefix-map=/home/sylvestre/dev/debian/pkg-llvm/llvm-toolchain/branches/llvm-toolchain-15-15.0.0~+rc3/= -no-canonical-prefixes -O3 -DNDEBUG -Wl,-z,relro -Wl,--build-id -fuse-ld=gold    -Wl,--gc-sections tools/bolt/tools/heatmap/CMakeFiles/llvm-bolt-heatmap.dir/heatmap.cpp.o -o bin/llvm-bolt-heatmap  -Wl,-rpath,"\$ORIGIN/../lib"  lib/libLLVMBOLTProfile.a  lib/libLLVMBOLTRewrite.a  lib/libLLVMBOLTUtils.a  lib/libLLVMBOLTProfile.a  lib/libLLVMBOLTRuntimeLibs.a  lib/libLLVMBOLTPasses.a  lib/libLLVMBOLTCore.a  lib/libLLVMBOLTUtils.a  lib/libLLVMBOLTTargetAArch64.a  lib/libLLVMBOLTTargetX86.a  lib/libLLVM-15.so.1 && :
lib/libLLVMBOLTTargetAArch64.a(AArch64MCPlusBuilder.cpp.o):AArch64MCPlusBuilder.cpp:function (anonymous namespace)::AArch64MCPlusBuilder::getTargetExprFor(llvm::MCInst&, llvm::MCExpr const*, llvm::MCContext&, unsigned long) const: error: undefined reference to 'llvm::AArch64MCExpr::create(llvm::MCExpr const*, llvm::AArch64MCExpr::VariantKind, llvm::MCContext&)'
lib/libLLVMBOLTTargetAArch64.a(AArch64MCPlusBuilder.cpp.o):AArch64MCPlusBuilder.cpp:function (anonymous namespace)::AArch64MCPlusBuilder::createLongJmp(std::vector<llvm::MCInst, std::allocator<llvm::MCInst> >&, llvm::MCSymbol const*, llvm::MCContext*, bool): error: undefined reference to 'llvm::AArch64MCExpr::create(llvm::MCExpr const*, llvm::AArch64MCExpr::VariantKind, llvm::MCContext&)'
[...]

On a debian linux amd64

[llvmbot]

@llvm/issue-subscribers-bolt

[aaupov]

Can you please provide the CMake invocation and compiler/linker versions?
Couldn't reproduce on Ubuntu 20.04 on tags/llvmorg-15.0.0-rc3.

[sylvestre]

This is the debian packages:

 cmake -S llvm/ -B build-llvm \
-G Ninja \
-DCMAKE_INSTALL_PREFIX=/usr/lib/llvm-15 \
-DLLVM_VERSION_SUFFIX= \
-DCMAKE_SUPPRESS_REGENERATION=ON \
-DCMAKE_BUILD_TYPE=Release \
-DCMAKE_C_FLAGS="-fstack-protector-strong -Wformat -Werror=format-security -Wno-unused-command-line-argument -Wdate-time -D_FORTIFY_SOURCE=2" \
-DCMAKE_CXX_FLAGS="-fstack-protector-strong -Wformat -Werror=format-security -Wno-unused-command-line-argument -Wdate-time -D_FORTIFY_SOURCE=2" \
-DCMAKE_SHARED_LINKER_FLAGS="-Wl,-z,relro -Wl,--build-id" \
-DCMAKE_MODULE_LINKER_FLAGS="-Wl,-z,relro -Wl,--build-id" \
-DCMAKE_EXE_LINKER_FLAGS="-Wl,-z,relro -Wl,--build-id" \
-DPACKAGE_VENDOR=Debian \
-DENABLE_LINKER_BUILD_ID=ON \
-DLLVM_TARGETS_TO_BUILD=Native \
-DLLVM_ENABLE_PROJECTS="clang;clang-tools-extra;lld;cross-project-tests;mlir;bolt;polly;lldb" \
-DLLVM_ENABLE_RUNTIMES="compiler-rt;libcxx;libcxxabi;libunwind;openmp" \
-DLLVM_ENABLE_PIC=ON \
-DLLVM_ENABLE_RTTI=ON \
-DLLVM_BUILD_DOCS=OFF \
-DLLVM_INCLUDE_GO_TESTS=OFF \
-DLLVM_USE_RELATIVE_PATHS_IN_FILES=ON \
-DLLVM_TEMPORARILY_ALLOW_OLD_TOOLCHAIN=ON \
-DLLVM_ENABLE_CURL=ON \
-DCLANG_PLUGIN_SUPPORT=OFF \
-DCLANG_BUILD_EXAMPLES=OFF \
-DCLANG_DEFAULT_LINKER=ld \
-DCLANG_DEFAULT_CXX_STDLIB=libstdc++ \
-DCLANG_DEFAULT_RTLIB=libgcc \
-DCOMPILER_RT_BUILD_XRAY=OFF \
-DCOMPILER_RT_INCLUDE_TESTS=OFF \
-DCOMPILER_RT_USE_LIBCXX=OFF \
-DCOMPILER_RT_USE_BUILTINS_LIBRARY=ON \
-DLIBUNWIND_USE_COMPILER_RT=ON \
-DLIBUNWIND_INSTALL_LIBRARY=OFF \
-DLIBCXXABI_ENABLE_EXCEPTIONS=ON \
-DLIBCXXABI_USE_COMPILER_RT=ON \
-DLIBCXXABI_INSTALL_LIBRARY=OFF \
-DLIBCXX_ENABLE_EXCEPTIONS=ON \
-DLIBCXX_USE_COMPILER_RT=ON \
-DLIBCXX_ENABLE_STATIC_ABI_LIBRARY=ON \
-DLIBCXX_STATICALLY_LINK_ABI_IN_STATIC_LIBRARY=ON \
-DLIBCXX_STATICALLY_LINK_ABI_IN_SHARED_LIBRARY=OFF \
-DLIBCXX_INSTALL_LIBRARY=OFF \
-DOPENMP_ENABLE_LIBOMP_PROFILING=OFF \
-DOPENMP_ENABLE_LIBOMPTARGET_PROFILING=OFF \
-DLIBOMP_ENABLE_RTTI=OFF \
-DLIBOMP_OMPT_SUPPORT=OFF \
 -DLLVM_USE_LINKER=gold -DLLVM_BINUTILS_INCDIR=/usr/include/ -DLIBCXXABI_USE_LLVM_UNWINDER=ON -DLIBOMP_LIBFLAGS="-lm" -DOPENMP_USE_LLVM_UNWINDER=ON -DMLIR_INSTALL_AGGREGATE_OBJECTS=OFF -DLLDB_ENABLE_LUA=OFF \
-DBUILTINS_CMAKE_ARGS="-DCMAKE_C_FLAGS=-fstack-protector-strong -Wformat -Werror=format-security -Wno-unused-command-line-argument -Wdate-time -D_FORTIFY_SOURCE=2;-DCMAKE_CXX_FLAGS=-fstack-protector-strong -Wformat -Werror=format-security -Wno-unused-command-line-argument -Wdate-time -D_FORTIFY_SOURCE=2;-DCMAKE_EXE_LINKER_FLAGS=-Wl,-z,relro -Wl,--build-id;-DCMAKE_SHARED_LINKER_FLAGS=-Wl,-z,relro -Wl,--build-id;-DCMAKE_MODULE_LINKER_FLAGS=-Wl,-z,relro -Wl,--build-id;-DCMAKE_BUILD_TYPE=Release;-DLLVM_ENABLE_PER_TARGET_RUNTIME_DIR=OFF" \
-DRUNTIMES_CMAKE_ARGS="-DCMAKE_C_FLAGS=-fstack-protector-strong -Wformat -Werror=format-security -Wno-unused-command-line-argument -Wdate-time -D_FORTIFY_SOURCE=2;-DCMAKE_CXX_FLAGS=-fstack-protector-strong -Wformat -Werror=format-security -Wno-unused-command-line-argument -Wdate-time -D_FORTIFY_SOURCE=2;-DCMAKE_EXE_LINKER_FLAGS=-Wl,-z,relro -Wl,--build-id;-DCMAKE_SHARED_LINKER_FLAGS=-Wl,-z,relro -Wl,--build-id;-DCMAKE_MODULE_LINKER_FLAGS=-Wl,-z,relro -Wl,--build-id;-DCMAKE_BUILD_TYPE=Release;-DLLVM_ENABLE_PER_TARGET_RUNTIME_DIR=OFF" \
-DCLANG_ENABLE_BOOTSTRAP=ON \
-DCLANG_BOOTSTRAP_TARGETS="check-all;check-llvm;check-clang;check-clang-tools;check-lld;check-libcxx;check-libcxxabi;check-mlir;check-sanitizer;llvm-config;test-suite" \
-DCLANG_BOOTSTRAP_PASSTHROUGH="CMAKE_INSTALL_PREFIX;CMAKE_SUPPRESS_REGENERATION;ENABLE_LINKER_BUILD_ID;LLVM_ENABLE_PIC;LLVM_ENABLE_RTTI;LLVM_INCLUDE_GO_TESTS;LLVM_USE_RELATIVE_PATHS_IN_FILES;CLANG_DEFAULT_LINKER;CLANG_DEFAULT_CXX_STDLIB;CLANG_DEFAULT_RTLIB;COMPILER_RT_USE_LIBCXX;COMPILER_RT_USE_BUILTINS_LIBRARY;COMPILER_RT_INCLUDE_TESTS;LIBUNWIND_USE_COMPILER_RT;LIBCXXABI_ENABLE_EXCEPTIONS;LIBCXXABI_USE_COMPILER_RT;LIBCXX_USE_COMPILER_RT;LIBCXX_ENABLE_EXCEPTIONS;LIBCXX_ENABLE_STATIC_ABI_LIBRARY;LIBCXX_STATICALLY_LINK_ABI_IN_STATIC_LIBRARY;LIBCXX_STATICALLY_LINK_ABI_IN_SHARED_LIBRARY;LIBOMP_ENABLE_RTTI;LLVM_TEMPORARILY_ALLOW_OLD_TOOLCHAIN" \
-DBOOTSTRAP_CMAKE_BUILD_TYPE=RelWithDebInfo \
-DBOOTSTRAP_CMAKE_C_FLAGS_RELWITHDEBINFO="-O2 -DNDEBUG -g1" \
-DBOOTSTRAP_CMAKE_CXX_FLAGS_RELWITHDEBINFO="-O2 -DNDEBUG -g1" \
-DBOOTSTRAP_CMAKE_C_FLAGS="-fstack-protector-strong -Wformat -Werror=format-security -Wno-unused-command-line-argument -Wdate-time -D_FORTIFY_SOURCE=2" \
-DBOOTSTRAP_CMAKE_CXX_FLAGS="-fstack-protector-strong -Wformat -Werror=format-security -Wno-unused-command-line-argument -Wdate-time -D_FORTIFY_SOURCE=2" \
-DBOOTSTRAP_CMAKE_SHARED_LINKER_FLAGS="-Wl,-z,relro -Wl,--build-id" \
-DBOOTSTRAP_CMAKE_MODULE_LINKER_FLAGS="-Wl,-z,relro -Wl,--build-id" \
-DBOOTSTRAP_CMAKE_EXE_LINKER_FLAGS="-Wl,-z,relro -Wl,--build-id" \
-DBOOTSTRAP_LLVM_ENABLE_FFI=ON \
-DBOOTSTRAP_LLVM_ENABLE_DUMP=ON \
-DBOOTSTRAP_LLVM_ENABLE_LIBPFM=ON \
-DBOOTSTRAP_LLVM_ENABLE_SPHINX=ON \
-DBOOTSTRAP_CLANG_DEFAULT_PIE_ON_LINUX=ON \
-DBOOTSTRAP_SPHINX_WARNINGS_AS_ERRORS=OFF \
-DBOOTSTRAP_LLVM_USE_RELATIVE_PATHS_IN_FILES=ON \
-DBOOTSTRAP_LLVM_INSTALL_UTILS=ON \
-DBOOTSTRAP_LLVM_VERSION_SUFFIX= \
-DBOOTSTRAP_LLVM_POLLY_LINK_INTO_TOOLS=ON \
-DBOOTSTRAP_LLVM_EXPERIMENTAL_TARGETS_TO_BUILD="M68k" \
-DBOOTSTRAP_LLVM_LINK_LLVM_DYLIB=ON \
    -DBOOTSTRAP_LLVM_ENABLE_CURL=ON \
-DBOOTSTRAP_CLANG_LINK_CLANG_DYLIB=ON \
-DBOOTSTRAP_LIBCLANG_LIBRARY_VERSION=1 \
-DBOOTSTRAP_LIBCXX_INSTALL_EXPERIMENTAL_LIBRARY=ON \
-DBOOTSTRAP_PYTHON_EXECUTABLE=/usr/bin/python3 \
 -DBOOTSTRAP_LLVM_ENABLE_Z3_SOLVER=ON  -DBOOTSTRAP_CLANG_SYSTEMZ_DEFAULT_ARCH=z196  -DBOOTSTRAP_CLANGD_ENABLE_REMOTE=ON  -DBOOTSTRAP_LLVM_USE_PERF=ON  -DBOOTSTRAP_LLVM_ENABLE_LIBPFM=ON  -DBOOTSTRAP_LLVM_USE_LINKER=gold  -DBOOTSTRAP_LLVM_BINUTILS_INCDIR=/usr/include/  -DBOOTSTRAP_LIBCXXABI_USE_LLVM_UNWINDER=ON  -DBOOTSTRAP_LIBOMP_LIBFLAGS="-lm"  -DBOOTSTRAP_OPENMP_USE_LLVM_UNWINDER=ON  -DBOOTSTRAP_MLIR_INSTALL_AGGREGATE_OBJECTS=OFF  -DBOOTSTRAP_LLDB_ENABLE_LUA=OFF \
-DBOOTSTRAP_BUILTINS_CMAKE_ARGS="-DCMAKE_C_FLAGS=-fstack-protector-strong -Wformat -Werror=format-security -Wno-unused-command-line-argument -Wdate-time -D_FORTIFY_SOURCE=2;-DCMAKE_CXX_FLAGS=-fstack-protector-strong -Wformat -Werror=format-security -Wno-unused-command-line-argument -Wdate-time -D_FORTIFY_SOURCE=2;-DCMAKE_EXE_LINKER_FLAGS=-Wl,-z,relro -Wl,--build-id;-DCMAKE_SHARED_LINKER_FLAGS=-Wl,-z,relro -Wl,--build-id;-DCMAKE_MODULE_LINKER_FLAGS=-Wl,-z,relro -Wl,--build-id;-DCMAKE_BUILD_TYPE=RelWithDebInfo;-DCMAKE_C_FLAGS_RELWITHDEBINFO=-O2 -DNDEBUG -g1;-DCMAKE_CXX_FLAGS_RELWITHDEBINFO=-O2 -DNDEBUG -g1;-DLLVM_ENABLE_PER_TARGET_RUNTIME_DIR=OFF" \
-DBOOTSTRAP_RUNTIMES_CMAKE_ARGS="-DCMAKE_C_FLAGS=-fstack-protector-strong -Wformat -Werror=format-security -Wno-unused-command-line-argument -Wdate-time -D_FORTIFY_SOURCE=2;-DCMAKE_CXX_FLAGS=-fstack-protector-strong -Wformat -Werror=format-security -Wno-unused-command-line-argument -Wdate-time -D_FORTIFY_SOURCE=2;-DCMAKE_EXE_LINKER_FLAGS=-Wl,-z,relro -Wl,--build-id -L/extra-dev/branches/llvm-toolchain-15-15.0.0~+rc3/build-llvm/lib;-DCMAKE_SHARED_LINKER_FLAGS=-Wl,-z,relro -Wl,--build-id -L/extra-dev/branches/llvm-toolchain-15-15.0.0~+rc3/build-llvm/lib;-DCMAKE_MODULE_LINKER_FLAGS=-Wl,-z,relro -Wl,--build-id -L/extra-dev/branches/llvm-toolchain-15-15.0.0~+rc3/build-llvm/lib;-DCMAKE_BUILD_TYPE=RelWithDebInfo;-DCMAKE_C_FLAGS_RELWITHDEBINFO=-O2 -DNDEBUG -g1;-DCMAKE_CXX_FLAGS_RELWITHDEBINFO=-O2 -DNDEBUG -g1;-DLLVM_ENABLE_PER_TARGET_RUNTIME_DIR=OFF"

[aaupov]

I can't reproduce the issue using cmake command or by following https://apt.llvm.org/building-pkgs.php (+adding BOLT to the list of projects).
I vaguely understand I need a branch llvm-toolchain-15-15.0.0~+rc3 at https://salsa.debian.org/pkg-llvm-team/llvm-toolchain to repro exactly. Can you please push the branch for me?

[aaupov]

Alternatively, please try this out:

diff --git a/bolt/lib/Target/AArch64/CMakeLists.txt b/bolt/lib/Target/AArch64/CMakeLists.txt
index 96c70168196e..f9767179dcb3 100644
--- a/bolt/lib/Target/AArch64/CMakeLists.txt
+++ b/bolt/lib/Target/AArch64/CMakeLists.txt
@@ -2,7 +2,7 @@ set(LLVM_LINK_COMPONENTS
   BOLTCore
   MC
   Support
-  AArch64Desc
+  AArch64
   )

 add_llvm_library(LLVMBOLTTargetAArch64

Or revert 64421e1 which I tested as thoroughly as I could but apparently failed to account for some build configurations.

[sylvestre]

Alternatively, please try this out:

diff --git a/bolt/lib/Target/AArch64/CMakeLists.txt b/bolt/lib/Target/AArch64/CMakeLists.txt
index 96c70168196e..f9767179dcb3 100644
--- a/bolt/lib/Target/AArch64/CMakeLists.txt
+++ b/bolt/lib/Target/AArch64/CMakeLists.txt
@@ -2,7 +2,7 @@ set(LLVM_LINK_COMPONENTS
   BOLTCore
   MC
   Support
-  AArch64Desc
+  AArch64
   )

 add_llvm_library(LLVMBOLTTargetAArch64

Or revert 64421e1 which I tested as thoroughly as I could but apparently failed to account for some build configurations.

Both won't work.
Adding these libs by hand fixes the issue:
lib/libLLVMBOLTTargetAArch64.a lib/libLLVMBOLTTargetX86.a lib/libLLVMAArch64Desc.a lib/libLLVMAArch64Info.a lib/libLLVMAArch64Utils.a lib/libLLVMX86Desc.a lib/libLLVMX86Info. but dunno where to add them
Still on bin/llvm-bolt-heatmap

To reproduce:

dget -x https://apt.llvm.org/unstable/pool/main/l/llvm-toolchain-15/llvm-toolchain-15_15.0.0~%2b%2b20220905103056%2b4ba6a9c9f65b-1~exp1~20220905103153.50.dsc.asc
cd llvm-toolchain-15*
# edit debian/rules to built bolt

And apply

Index: llvm-toolchain-15-15.0.0~+rc3/bolt/tools/merge-fdata/CMakeLists.txt
===================================================================
--- llvm-toolchain-15-15.0.0~+rc3.orig/bolt/tools/merge-fdata/CMakeLists.txt
+++ llvm-toolchain-15-15.0.0~+rc3/bolt/tools/merge-fdata/CMakeLists.txt
@@ -9,6 +9,6 @@ add_bolt_tool(merge-fdata
 set_target_properties(merge-fdata PROPERTIES FOLDER "BOLT")
 
 # Emit relocations for BOLT meta test (bolt/test/runtime/meta-merge-fdata.test)
-if (BOLT_INCLUDE_TESTS AND UNIX AND NOT APPLE)
-  target_link_options(merge-fdata PRIVATE LINKER:--emit-relocs)
-endif()
+# if (BOLT_INCLUDE_TESTS AND UNIX AND NOT APPLE)
+#   target_link_options(merge-fdata PRIVATE LINKER:--emit-relocs)
+# endif()

[aaupov]

Alternatively, please try this out:

diff --git a/bolt/lib/Target/AArch64/CMakeLists.txt b/bolt/lib/Target/AArch64/CMakeLists.txt
index 96c70168196e..f9767179dcb3 100644
--- a/bolt/lib/Target/AArch64/CMakeLists.txt
+++ b/bolt/lib/Target/AArch64/CMakeLists.txt
@@ -2,7 +2,7 @@ set(LLVM_LINK_COMPONENTS
   BOLTCore
   MC
   Support
-  AArch64Desc
+  AArch64
   )

 add_llvm_library(LLVMBOLTTargetAArch64

Or revert 64421e1 which I tested as thoroughly as I could but apparently failed to account for some build configurations.

Both won't work. Adding these libs by hand fixes the issue: lib/libLLVMBOLTTargetAArch64.a lib/libLLVMBOLTTargetX86.a lib/libLLVMAArch64Desc.a lib/libLLVMAArch64Info.a lib/libLLVMAArch64Utils.a lib/libLLVMX86Desc.a lib/libLLVMX86Info. but dunno where to add them Still on bin/llvm-bolt-heatmap

To reproduce:

dget -x https://apt.llvm.org/unstable/pool/main/l/llvm-toolchain-15/llvm-toolchain-15_15.0.0~%2b%2b20220905103056%2b4ba6a9c9f65b-1~exp1~20220905103153.50.dsc.asc
cd llvm-toolchain-15*
# edit debian/rules to built bolt

Sorry, I'm not very familiar with dget. dget downloads the asc file with PGP but then nothing happens. Do I need to add some sources?

[sylvestre]

sorry, remove the .asc :)

[aaupov]

Reproduced locally.

[aaupov]

The issue is probably that LLVM libraries specified with LLVM_LINK_COMPONENTS are not passed to the linker in stage2 build.

excerpt from the failing compiler invocation:

lib/libLLVMBOLTProfile.a lib/libLLVMBOLTRewrite.a lib/libLLVMBOLTUtils.a lib/libLLVMBOLTProfile.a lib/libLLVMBOLTRuntimeLibs.a lib/libLLVMBOLTPasses.a lib/libLLVMBOLTCore.a lib/libLLVMBOLTUtils.a lib/libLLVMBOLTTargetAArch64.a lib/libLLVMBOLTTargetX86.a lib/libLLVM-15.so.1

There should also be libLLVMX86 and libLLVMAArch64 libraries.

Since it's not an issue for a regular build, I would assume they're passed just fine. Still trying to understand how to fix this issue.

[aaupov]

Quick note: the issue doesn't reproduce with a regular two-stage build (-DCLANG_ENABLE_BOOTSTRAP=On)