bazel links program using clang rather than clang++, incompatible with -fsanitize=undefined (UBSan)

[trittweiler]

Description of the problem / feature request:

Linking a C++ program with clang -fsanitize=undefined results in undefined symbols.
It is necessary to use clang++ -fsanitize=undefined instead.

And bazel as of 3.7.2 tries to link C++ object files using clang rather than clang++.

What operating system are you running Bazel on?

Ubuntu Focal 20.04.1 LTS.

The test case was done using clang 10.0.0-4ubuntu1, that's simply from the clang-10 Ubuntu package.

What's the output of bazel info release?

release 3.7.2

Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

Here's a test case:

// ubsan_test.cc
#include <iostream>

int main(int argc, char **argv) {
  std::cout << "program " << argv[0] << " started" << std::endl;
  int k = 0x7fffffff;
  k += argc;
  return 0;
}

# BUILD
cc_binary(
    name = "ubsan_test",
    srcs = [ "ubsan_test.cc" ]
)

# WORKSPACE

And we now try to build, link and run :ubsan_test as follows:

> bazel run :ubsan_test -c dbg --repo_env=CC=clang --repo_env=CXX=clang++ --copt -fsanitize=undefined --copt -O1 --copt -fno-omit-frame-pointer --linkopt -fsanitize=undefined --subcommands
Starting local Bazel server and connecting to it...
INFO: Analyzed target //:ubsan_test (15 packages loaded, 53 targets configured).
INFO: Found 1 target...
SUBCOMMAND: # //:ubsan_test [action 'Compiling ubsan_test.cc', configuration: 9ba3d5284ce70f920a40e5da377608177842fbf5b6c778ffecc875296ea059e1, execution platform: @local_config_platform//:host]
(cd /home/trittweiler/.cache/bazel/_bazel_trittweiler/ce7c0737658cab18554c03aaea49c3e0/execroot/__main__ && \
  exec env - \
    PATH=/home/trittweiler/.local/bin:/home/trittweiler/.local/software/yED/jre/bin:/home/trittweiler/.local/software/watchman/bin:/home/trittweiler/.local/software/rtags/bin:/home/trittweiler/.local/software/gopls/bin:/home/trittweiler/.local/software/emacs-git/bin:/home/trittweiler/.local/software/buck/bin:/opt/intellij-idea-community/bin:/opt/containerd/bin:/opt/android-studio/bin:/home/trittweiler/software/mtr/sbin/:/home/trittweiler/software/youtube-dl/bin/:/home/trittweiler/software/stumpwm/bin/:/home/trittweiler/software/sbcl/bin/:/home/trittweiler/software/cutter/bin/:/home/trittweiler/software/ccl-1.11.5/bin/:/home/trittweiler/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin \
    PWD=/proc/self/cwd \
  /usr/bin/clang -U_FORTIFY_SOURCE -fstack-protector -Wall -Wthread-safety -Wself-assign -fcolor-diagnostics -fno-omit-frame-pointer -g '-std=c++0x' -MD -MF bazel-out/k8-dbg/bin/_objs/ubsan_test/ubsan_test.pic.d '-frandom-seed=bazel-out/k8-dbg/bin/_objs/ubsan_test/ubsan_test.pic.o' -fPIC -iquote . -iquote bazel-out/k8-dbg/bin -iquote external/bazel_tools -iquote bazel-out/k8-dbg/bin/external/bazel_tools '-fsanitize=undefined' -O1 -fno-omit-frame-pointer -no-canonical-prefixes -Wno-builtin-macro-redefined '-D__DATE__="redacted"' '-D__TIMESTAMP__="redacted"' '-D__TIME__="redacted"' -c ubsan_test.cc -o bazel-out/k8-dbg/bin/_objs/ubsan_test/ubsan_test.pic.o)
SUBCOMMAND: # //:ubsan_test [action 'Linking ubsan_test', configuration: 9ba3d5284ce70f920a40e5da377608177842fbf5b6c778ffecc875296ea059e1, execution platform: @local_config_platform//:host]
(cd /home/trittweiler/.cache/bazel/_bazel_trittweiler/ce7c0737658cab18554c03aaea49c3e0/execroot/__main__ && \
  exec env - \
    PATH=/home/trittweiler/.local/bin:/home/trittweiler/.local/software/yED/jre/bin:/home/trittweiler/.local/software/watchman/bin:/home/trittweiler/.local/software/rtags/bin:/home/trittweiler/.local/software/gopls/bin:/home/trittweiler/.local/software/emacs-git/bin:/home/trittweiler/.local/software/buck/bin:/opt/intellij-idea-community/bin:/opt/containerd/bin:/opt/android-studio/bin:/home/trittweiler/software/mtr/sbin/:/home/trittweiler/software/youtube-dl/bin/:/home/trittweiler/software/stumpwm/bin/:/home/trittweiler/software/sbcl/bin/:/home/trittweiler/software/cutter/bin/:/home/trittweiler/software/ccl-1.11.5/bin/:/home/trittweiler/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin \
    PWD=/proc/self/cwd \
  /usr/bin/clang @bazel-out/k8-dbg/bin/ubsan_test-2.params)
ERROR: /home/trittweiler/tmp/bazel-bug/BUILD:1:10: Linking of rule '//:ubsan_test' failed (Exit 1): clang failed: error executing command /usr/bin/clang @bazel-out/k8-dbg/bin/ubsan_test-2.params

Use --sandbox_debug to see verbose messages from the sandbox clang failed: error executing command /usr/bin/clang @bazel-out/k8-dbg/bin/ubsan_test-2.params

Use --sandbox_debug to see verbose messages from the sandbox
ubsan_test.cc:4: error: undefined reference to '__ubsan_vptr_type_cache'
ubsan_test.cc:4: error: undefined reference to '__ubsan_handle_dynamic_type_cache_miss'
/usr/bin/../lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/bits/locale_facets.h:874: error: undefined reference to '__ubsan_vptr_type_cache'
/usr/bin/../lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/bits/locale_facets.h:874: error: undefined reference to '__ubsan_handle_dynamic_type_cache_miss'
/usr/bin/../lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/bits/locale_facets.h:875: error: undefined reference to '__ubsan_handle_dynamic_type_cache_miss'
/usr/bin/../lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/bits/locale_facets.h:876: error: undefined reference to '__ubsan_handle_dynamic_type_cache_miss'
clang: error: linker command failed with exit code 1 (use -v to see invocation)
Target //:ubsan_test failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 6.665s, Critical Path: 0.43s
INFO: 8 processes: 7 internal, 1 linux-sandbox.
FAILED: Build did NOT complete successfully
FAILED: Build did NOT complete successfully

Now I am going to manually repeat the link step (the /usr/bin/clang @bazel-out/k8-dbg/bin/ubsan_test-2.params) but replaceclang with clang++:

> (cd /home/trittweiler/.cache/bazel/_bazel_trittweiler/ce7c0737658cab18554c03aaea49c3e0/execroot/__main__ && \
>   exec env - \
>     PATH=/home/trittweiler/.local/bin:/home/trittweiler/.local/software/yED/jre/bin:/home/trittweiler/.local/software/watchman/bin:/home/trittweiler/.local/software/rtags/bin:/home/trittweiler/.local/software/gopls/bin:/home/trittweiler/.local/software/emacs-git/bin:/home/trittweiler/.local/software/buck/bin:/opt/intellij-idea-community/bin:/opt/containerd/bin:/opt/android-studio/bin:/home/trittweiler/software/mtr/sbin/:/home/trittweiler/software/youtube-dl/bin/:/home/trittweiler/software/stumpwm/bin/:/home/trittweiler/software/sbcl/bin/:/home/trittweiler/software/cutter/bin/:/home/trittweiler/software/ccl-1.11.5/bin/:/home/trittweiler/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin \
>     PWD=/proc/self/cwd \
>   /usr/bin/clang++ @bazel-out/k8-dbg/bin/ubsan_test-2.params)

And that succeeded just fine. And when I manually run the produced binary, it works just fine:

 UBSAN_OPTIONS=print_stacktrace=1 ./bazel-out/k8-dbg/bin/ubsan_test
program ./bazel-out/k8-dbg/bin/ubsan_test started
ubsan_test.cc:6:5: runtime error: signed integer overflow: 2147483647 + 1 cannot be represented in type 'int'
    #0 0x424f3c in main /proc/self/cwd/ubsan_test.cc:6:5
    #1 0x7f20290380b2 in __libc_start_main /build/glibc-ZN95T4/glibc-2.31/csu/../csu/libc-start.c:308:16
    #2 0x40352d in _start (/home/trittweiler/.cache/bazel/_bazel_trittweiler/ce7c0737658cab18554c03aaea49c3e0/execroot/__main__/bazel-out/k8-dbg/bin/ubsan_test+0x40352d)

[trittweiler]

Maybe I am just mistaken, and bazel does not actually look at the CXX environment variable?

It seems to be compiling the ubsan_test.cc also with clang, i.e. the contents of the CC variable.

So am I simply using it wrong?

[trittweiler]

Assuming bazel does indeed not look at CXX at this time, this issue is essentially evidence for someone tripping over what is covered by Issue #5186.

[chancila]

bazel currently compiles through the base clang/gcc driver and doesn't support the c++ drivers of these commands, bazel c++ toolchains generally have to be explicit about any implicit c++ stuff that would otherwise happen when you use the c++ drivers for clang/gcc.

if you run -v on your invocation with clang++ you'll probably see something like --whole-archive /usr/lib/clang/11.0.0/lib/linux/libclang_rt.ubsan_standalone_cxx-x86_64.a --no-whole-archive --dynamic-list=/usr/lib/clang/11.0.0/lib/linux/libclang_rt.ubsan_standalone_cxx-x86_64.a.syms

you effectively need to do this manually in your toolchain (or via --linkopts, or via the linkopts rule attributes), afaik this is a known limitation of the toolchain design, the link invocation is also always assumed to be c++ which isn't great but is typically benign because c code won't emit any references to c++ symbols.

[fmeum]

See #11122 (comment) for two easy ways to fix this problem. There is no need to hardcode library paths. Appending either

--linkopt -fsanitize-link-c++-runtime

or

--linkopt --driver-mode=g++ 

to the commandline above makes clang link the UBSan C++ runtime libraries without the need to use clang++. In fact, the latter flag makes clang behave as if it were clang++.

[matts1]

This workaround solves the problem of compiling C++ code with a C compiler by trying to act like a C++ compiler. I was successfully able to compile C++ code with this workaround, but doing so broke the compilation of some of our dependencies that use C (see below).

IMO, the fact that we need workarounds for this is evidence that we should allow specifying both the C and C++ compiler.

We get errors like:

bazel build @zstd//:zstd
INFO: Analyzed target @zstd//:zstd (2 packages loaded, 16111 targets configured).
INFO: Found 1 target...
ERROR: /usr/local/google/home/msta/.cache/bazel/_bazel_msta/ed8a9562ed8b7b4d013265ed31cbccd4/external/zstd/BUILD.bazel:48:10: Linking external/zstd/zstd failed: (Exit 1): x86_64-cros-linux-gnu-clang failed: error executing command (from target @zstd//:zstd) /usr/local/google/home/msta/.cache/bazel/_bazel_msta/ed8a9562ed8b7b4d013265ed31cbccd4/external/_main~toolchains~toolchain_sdk/bin/x86_64-cros-linux-gnu-clang ... (remaining 1 argument skipped)

Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging
ld.lld: error: undefined symbol: __cxa_begin_catch
>>> referenced by benchzstd.c
>>>               bazel-out/k8-dbg/bin/external/zstd/_objs/zstd/benchzstd.pic.o:(__clang_call_terminate)

ld.lld: error: undefined symbol: std::terminate()
>>> referenced by benchzstd.c
>>>               bazel-out/k8-dbg/bin/external/zstd/_objs/zstd/benchzstd.pic.o:(__clang_call_terminate)

ld.lld: error: undefined symbol: __gxx_personality_v0
>>> referenced by benchzstd.c
>>>               bazel-out/k8-dbg/bin/external/zstd/_objs/zstd/benchzstd.pic.o:(DW.ref.__gxx_personality_v0)
clang: error: linker command failed with exit code 1 (use -v to see invocation)
Target @zstd//:zstd failed to build

And hundreds of warnings like the following:

INFO: From Compiling lib/compress/zstd_lazy.c:
clang: warning: treating 'c' input as 'c++' when in C++ mode, this behavior is deprecated [-Wdeprecated]