Different linkers/flags used in cmake's generate and build steps.

[PiotrSikora]

This is a regression when using rules_foreign_cc with Bazel v5.0.0 (which for some reason uses lld in the generate step), and it works fine with Bazel v4.2.2 (which uses ld.gold in the generate step). FYI, @Wyverald.

However, the underlying issue seems to be with different flags being passed to generate and build steps in the cmake rule.

Possibly related? #235

Background

We've ran into this issue back in 2019, and we've been carrying a patch in Envoy that removes the check for the problematic flag (-Wl,--color-diagnostics) when building LLVM using rules_foreign_cc.

However, we've been building LLVM using rules_foreign_cc (using the same rules) without the patch in Proxy-Wasm C++ Host since early 2020, using various Bazel versions (v3.6.0 to v4.2.2).

Bazel v4.2.2

The generate step doesn't detect support for -Wl,--color-diagnostics in ld.gold:

.../HandleLLVMOptions.cmake(849):  check_linker_flag(-Wl,--color-diagnostics LINKER_SUPPORTS_COLOR_DIAGNOSTICS )
.../CheckLinkerFlag.cmake(4):  set(CMAKE_EXE_LINKER_FLAGS -fuse-ld=/usr/bin/ld.gold -Wl,-no-as-needed -Wl,-z,relro,-z,now -B/usr/lib/llvm-12/bin -lstdc++ -lm -lpthread -ldl -Wl,--color-diagnostics )
...
-- Performing Test LINKER_SUPPORTS_COLOR_DIAGNOSTICS
...
FAILED: cmTC_b9422
: && /usr/lib/llvm-12/bin/clang -U_FORTIFY_SOURCE -fstack-protector -Wall -Wthread-safety -Wself-assign -Wno-free-nonheap-object -fcolor-diagnostics -fno-omit-frame-pointer -std=c++0x -no-canonical-prefixes -Wno-builtin-macro-redefined -D__DATE__="redacted" -D__TIMESTAMP__="redacted" -D__TIME__="redacted" -std=c++17 -UDEBUG -Wno-unused-command-line-argument -fPIC -fvisibility-inlines-hidden -Werror=date-time -Werror=unguarded-availability-new -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -pedantic -Wno-long-long -Wimplicit-fallthrough -Wcovered-switch-default -Wno-noexcept-type -Wnon-virtual-dtor -Wdelete-non-virtual-dtor -Wsuggest-override -Wno-comment -Wstring-conversion -fdiagnostics-color  -Werror=unguarded-availability-new -fuse-ld=/usr/bin/ld.gold -Wl,-no-as-needed -Wl,-z,relro,-z,now -B/usr/lib/llvm-12/bin -lstdc++ -lm -lpthread -ldl -Wl,--color-diagnostics CMakeFiles/cmTC_b9422.dir/src.cxx.o -o cmTC_b9422  -lm && :
clang: warning: '-fuse-ld=' taking a path is deprecated. Use '--ld-path=' instead [-Wfuse-ld-path]
/usr/bin/ld.gold: --color-diagnostics: unknown option
/usr/bin/ld.gold: use the --help option for usage information
clang: error: linker command failed with exit code 1 (use -v to see invocation)
ninja: build stopped: subcommand failed.
...
-- Performing Test LINKER_SUPPORTS_COLOR_DIAGNOSTICS - Failed

The build step succeeds:

...
[1003/1598] : && /usr/lib/llvm-12/bin/clang -fPIC -U_FORTIFY_SOURCE -fstack-protector -Wall -Wthread-safety -Wself-assign -Wno-free-nonheap-object -fcolor-diagnostics -fno-omit-frame-pointer -std=c++0x -no-canonical-prefixes -Wno-builtin-macro-redefined -D__DATE__="redacted" -D__TIMESTAMP__="redacted" -D__TIME__="redacted" -std=c++17 -UDEBUG -Wno-unused-command-line-argument -fPIC -fvisibility-inlines-hidden -Werror=date-time -Werror=unguarded-availability-new -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -pedantic -Wno-long-long -Wimplicit-fallthrough -Wcovered-switch-default -Wno-noexcept-type -Wnon-virtual-dtor -Wdelete-non-virtual-dtor -Wsuggest-override -Wno-comment -Wstring-conversion -fdiagnostics-color -ffunction-sections -fdata-sections -O3 -DNDEBUG  -Wl,-z,nodelete   -Wl,-O3 -Wl,--gc-sections  -Wl,--version-script,".../proxy_wasm_cpp_host/bazel-out/k8-fastbuild/bin/external/llvm/llvm_lib.build_tmpdir/lib/Transforms/Hello/LLVMHello.exports" -shared  -o lib/LLVMHello.so lib/Transforms/Hello/CMakeFiles/LLVMHello.dir/Hello.cpp.o  -Wl,-rpath,"\$ORIGIN/../lib" && :
...

Note that the generate step was using -fuse-ld=/usr/bin/ld.gold, but there is no -fuse-ld option in the build step.

Bazel v5.0.0

The generate step detects support for -Wl,--color-diagnostics in lld:

.../HandleLLVMOptions.cmake(849):  check_linker_flag(-Wl,--color-diagnostics LINKER_SUPPORTS_COLOR_DIAGNOSTICS )
.../CheckLinkerFlag.cmake(4):  set(CMAKE_EXE_LINKER_FLAGS -fuse-ld=/usr/lib/llvm-12/bin/ld.lld -Wl,-no-as-needed -Wl,-z,relro,-z,now -B/usr/lib/llvm-12/bin -lstdc++ -lm -lpthread -ldl -Wl,--color-diagnostics )
...
-- Performing Test LINKER_SUPPORTS_COLOR_DIAGNOSTICS
...
[2/2] Linking CXX executable cmTC_8ea60
clang: warning: '-fuse-ld=' taking a path is deprecated. Use '--ld-path=' instead [-Wfuse-ld-path]
...
-- Performing Test LINKER_SUPPORTS_COLOR_DIAGNOSTICS - Success

The build step fails:

[1003/1598] Linking CXX shared module lib/LLVMHello.so
FAILED: lib/LLVMHello.so
: && /usr/lib/llvm-12/bin/clang -fPIC -U_FORTIFY_SOURCE -fstack-protector -Wall -Wthread-safety -Wself-assign -Wno-free-nonheap-object -fcolor-diagnostics -fno-omit-frame-pointer -std=c++0x -no-canonical-prefixes -Wno-builtin-macro-redefined -D__DATE__="redacted" -D__TIMESTAMP__="redacted" -D__TIME__="redacted" -std=c++17 -UDEBUG -Wno-unused-command-line-argument -fPIC -fvisibility-inlines-hidden -Werror=date-time -Werror=unguarded-availability-new -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -pedantic -Wno-long-long -Wimplicit-fallthrough -Wcovered-switch-default -Wno-noexcept-type -Wnon-virtual-dtor -Wdelete-non-virtual-dtor -Wsuggest-override -Wno-comment -Wstring-conversion -fdiagnostics-color -ffunction-sections -fdata-sections -O3 -DNDEBUG  -Wl,-z,nodelete -Wl,--color-diagnostics   -Wl,-O3 -Wl,--gc-sections  -Wl,--version-script,".../proxy_wasm_cpp_host/bazel-out/k8-fastbuild/bin/external/llvm/llvm_lib.build_tmpdir/lib/Transforms/Hello/LLVMHello.exports" -shared  -o lib/LLVMHello.so lib/Transforms/Hello/CMakeFiles/LLVMHello.dir/Hello.cpp.o  -Wl,-rpath,"\$ORIGIN/../lib" && :
/usr/bin/ld: unrecognized option '--color-diagnostics'
/usr/bin/ld: use the --help option for usage information
clang: error: linker command failed with exit code 1 (use -v to see invocation)

Note that the generate step was using -fuse-ld=/usr/lib/llvm-12/bin/ld.lld, but there is no -fuse-ld option in the build step.

Note that there is no -fuse-ld flag passed anywhere in the Bazel workspace, so I'm not sure why it selects lld.

Also, the issue persist even when using CC=gcc-11 CXX=g++-11, but then -fuse-ld=lld is used in the generate step.

Reproduce steps

The easiest way to reproduce this is to checkout my Proxy-Wasm C++ Host fork:

$ git clone https://github.com/PiotrSikora/proxy-wasm-cpp-host
$ cd proxy-wasm-cpp-host

Note that it currently uses rules_foreign_cc v0.2.0, but I also tested with the latest commit (50ee9979e60e8db38e10de45d2c60873a210bf55) and it makes no difference.

Checkout master with Bazel v4.2.2:

$ git checkout 44a63ebe58770a07e36ff1ff149f176fd8371810
$ bazel build @llvm//:llvm_lib

Checkout PR with Bazel v5.0.0:

$ git checkout 13d69463224dfa7f2aa4efdcc0d1155bb2f6730b
$ bazel build @llvm//:llvm_lib

Note: use generate_args = ["-GNinja", "--trace-expand"] and build_args = ["--verbose"] to see all the commands.

[github-actions]

This issue has been automatically marked as stale because it has not had any activity for 180 days. It will be closed if no further activity occurs in 30 days. Collaborators can add an assignee to keep this open indefinitely. Thanks for your contributions to rules_foreign_cc!

[github-actions]

This issue was automatically closed because it went 30 days without a reply since it was labeled "Can Close?"

[jiridanek]

I had to resort to compiling with bazel option

--linkopt="-fuse-ld=gold"

to get a successful build in my case. Without it, the linking command failed with

$ /usr/lib64/ccache/gcc -U_FORTIFY_SOURCE -fstack-protector -Wall -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer -fno-canonical-system-headers -Wno-builtin-macro-redefined -D__DATE__=\"redacted\" -D__TIMESTAMP__=\"redacted\" -D__TIME__=\"redacted\" -fvisibility=hidden -O3 -DNDEBUG -flto -fno-fat-lto-objects -fuse-ld=lld -Wl,-no-as-needed -Wl,-z,relro,-z,now -B/usr/lib64/ccache -pass-exit-codes -lstdc++ -lm "CMakeFiles/msgr-recv.dir/msgr-recv.c.o" "CMakeFiles/msgr-recv.dir/msgr-common.c.o" -o msgr-recv  -Wl,-rpath,/home/jdanek/.cache/bazel/_bazel_jdanek/36e30b937da1503cce68d3840f1df2bc/sandbox/linux-sandbox/17/execroot/_main/bazel-out/k8-fastbuild/bin/qpid-proton.build_tmpdir/c ../libqpid-proton.so.11.17.0 
ld.lld: error: undefined symbol: main
>>> referenced by /usr/lib/gcc/x86_64-redhat-linux/13/../../../../lib64/crt1.o:(_start)
collect2: error: ld returned 1 exit status

which apparently is caused by the combination of LTO and lld linker with gcc, as in -flto -fno-fat-lto-objects -fuse-ld=lld.