grpc Mac bazel build fails because of thread-local, but make passes

[vjpai]

Description of the problem / feature request / question:

I'm on the grpc/grpc project. We recently switched to using thread-local on one of our key structs. Although this project builds fine with make on Mac and Linux, it stopped building with bazel on Mac after we made this switch. Looking into it, it seems like tensorflow/serving also has the same issue.

If possible, provide a minimal example to reproduce the problem:

$ git clone git@github.com:grpc/grpc
$ cd grpc
$ bazel build //:grpc

Environment info

• Operating System: MacOS Sierra 10.12.6
• Bazel version (output of bazel info release): release 0.8.1-homebrew

Have you found anything relevant by searching the web?

Related issues are tensorflow/serving#1 and grpc/grpc#13856

Anything else, information or logs or outputs that would be helpful?

$ bazel build --verbose_failures //:grpc
INFO: Analysed target //:grpc (0 packages loaded).
INFO: Found 1 target...
ERROR: /Users/vpai/Git/grpc/BUILD:224:1: Linking of rule '//:grpc' failed (Exit 1): cc_wrapper.sh failed: error executing command 
  (cd /private/var/tmp/_bazel_vpai/5285458b308b3aadd65cb54a5ac76b0c/execroot/com_github_grpc_grpc && \
  exec env - \
    APPLE_SDK_PLATFORM=MacOSX \
    APPLE_SDK_VERSION_OVERRIDE=10.13 \
    PATH=/Users/vpai/google-cloud-sdk/bin:/usr/local/git/current/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin:/usr/local/go/bin \
    TMPDIR=/var/folders/xd/2k15ssh10lz6088_k_2lddtw007xtq/T/ \
    XCODE_VERSION_OVERRIDE=9.1.0 \
  external/local_config_cc/cc_wrapper.sh -fobjc-link-runtime -Wl,-S -shared -o bazel-out/darwin-fastbuild/bin/libgrpc.so bazel-out/darwin-fastbuild/bin/_objs/grpc/src/core/lib/surface/init.o bazel-out/darwin-fastbuild/bin/_objs/grpc/src/core/plugin_registry/grpc_plugin_registry.o -pthread -headerpad_max_install_names -lc++ -no-canonical-prefixes -undefined dynamic_lookup)

Use --sandbox_debug to see verbose messages from the sandbox
clang: warning: argument unused during compilation: '-pthread' [-Wunused-command-line-argument]
ld: illegal thread local variable reference to regular symbol __ZN9grpc_core7ExecCtx9exec_ctx_E for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
Target //:grpc failed to build
INFO: Elapsed time: 0.504s, Critical Path: 0.31s
FAILED: Build did NOT complete successfully

[hlopko]

Can you isolate the difference between your make and bazel link invocation? You can execute bazel with --subcommands to emit all command lines. You can then compare which options are different between your make build and bazel build.

[vjpai]

Hi there! Thanks for the response. Here's an example command from make that actually builds the relevant .o; note that generating the .a is just a matter of libtool in that case:

c++ -Ithird_party/protobuf/src -Ithird_party/googletest/googletest/include -Ithird_party/googletest/googlemock/include -Ithird_party/boringssl/include -Ithird_party/cares -Ithird_party/cares/cares -g -Wall -Wextra -Werror -Wno-long-long -Wno-unused-parameter -DOSATOMIC_USE_INLINED=1 -O2 -fPIC -I. -Iinclude -I/Users/vpai/Git/grpc/gens -I/usr/local/include -DNDEBUG -DINSTALL_PREFIX=\"/usr/local\"    -std=c++11 -stdlib=libc++ -fno-exceptions  -fno-rtti -fno-exceptions -MMD -MF /Users/vpai/Git/grpc/objs/opt/src/core/lib/iomgr/exec_ctx.dep -c -o /Users/vpai/Git/grpc/objs/opt/src/core/lib/iomgr/exec_ctx.o src/core/lib/iomgr/exec_ctx.cc

Here's the result of finding the version from the c++ command:

$ c++ -v
Apple LLVM version 9.0.0 (clang-900.0.38)
Target: x86_64-apple-darwin16.7.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin

And here's what I get from bazel in building the object file (which is fine):

SUBCOMMAND: # //:grpc_base_c [action 'Compiling src/core/lib/iomgr/exec_ctx.cc']
(cd /private/var/tmp/_bazel_vpai/c6cc7641f7bf8c8335597635bcca2d28/execroot/com_github_grpc_grpc && \
  exec env - \
    APPLE_SDK_PLATFORM=MacOSX \
    APPLE_SDK_VERSION_OVERRIDE=10.13 \
    PATH=/Users/vpai/google-cloud-sdk/bin:/usr/local/git/current/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin:/usr/local/go/bin \
    TMPDIR=/var/folders/xd/2k15ssh10lz6088_k_2lddtw007xtq/T/ \
    XCODE_VERSION_OVERRIDE=9.1.0 \
  external/local_config_cc/wrapped_clang '-D_FORTIFY_SOURCE=1' -fstack-protector -fcolor-diagnostics -Wall -Wthread-safety -Wself-assign -fno-omit-frame-pointer -O0 -DDEBUG '-std=c++11' -iquote . -iquote bazel-out/darwin-fastbuild/genfiles -iquote external/bazel_tools -iquote bazel-out/darwin-fastbuild/genfiles/external/bazel_tools -iquote external/com_google_absl -iquote bazel-out/darwin-fastbuild/genfiles/external/com_google_absl -iquote external/com_github_madler_zlib -iquote bazel-out/darwin-fastbuild/genfiles/external/com_github_madler_zlib -isystem include -isystem bazel-out/darwin-fastbuild/genfiles/include -isystem external/bazel_tools/tools/cpp/gcc3 -isystem external/com_github_madler_zlib -isystem bazel-out/darwin-fastbuild/genfiles/external/com_github_madler_zlib -MD -MF bazel-out/darwin-fastbuild/bin/_objs/grpc_base_c/src/core/lib/iomgr/exec_ctx.d '-frandom-seed=bazel-out/darwin-fastbuild/bin/_objs/grpc_base_c/src/core/lib/iomgr/exec_ctx.o' -D__CLANG_SUPPORT_DYN_ANNOTATION__ '-isysroot __BAZEL_XCODE_SDKROOT__' -no-canonical-prefixes -Wno-builtin-macro-redefined '-D__DATE__="redacted"' '-D__TIMESTAMP__="redacted"' '-D__TIME__="redacted"' -c src/core/lib/iomgr/exec_ctx.cc -o bazel-out/darwin-fastbuild/bin/_objs/grpc_base_c/src/core/lib/iomgr/exec_ctx.o)

And then what I get from bazel in actually doing the library build:

SUBCOMMAND: # //:grpc [action 'Linking libgrpc.so']
(cd /private/var/tmp/_bazel_vpai/c6cc7641f7bf8c8335597635bcca2d28/execroot/com_github_grpc_grpc && \
  exec env - \
    APPLE_SDK_PLATFORM=MacOSX \
    APPLE_SDK_VERSION_OVERRIDE=10.13 \
    PATH=/Users/vpai/google-cloud-sdk/bin:/usr/local/git/current/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin:/usr/local/go/bin \
    TMPDIR=/var/folders/xd/2k15ssh10lz6088_k_2lddtw007xtq/T/ \
    XCODE_VERSION_OVERRIDE=9.1.0 \
  external/local_config_cc/cc_wrapper.sh -fobjc-link-runtime -Wl,-S -shared -o bazel-out/darwin-fastbuild/bin/libgrpc.so bazel-out/darwin-fastbuild/bin/_objs/grpc/src/core/lib/surface/init.o bazel-out/darwin-fastbuild/bin/_objs/grpc/src/core/plugin_registry/grpc_plugin_registry.o -pthread -headerpad_max_install_names -lc++ -no-canonical-prefixes -undefined dynamic_lookup)
ERROR: /Users/vpai/Git/grpc-bazel-ready/BUILD:224:1: Linking of rule '//:grpc' failed (Exit 1): cc_wrapper.sh failed: error executing command 
  (cd /private/var/tmp/_bazel_vpai/c6cc7641f7bf8c8335597635bcca2d28/execroot/com_github_grpc_grpc && \
  exec env - \
    APPLE_SDK_PLATFORM=MacOSX \
    APPLE_SDK_VERSION_OVERRIDE=10.13 \
    PATH=/Users/vpai/google-cloud-sdk/bin:/usr/local/git/current/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin:/usr/local/go/bin \
    TMPDIR=/var/folders/xd/2k15ssh10lz6088_k_2lddtw007xtq/T/ \
    XCODE_VERSION_OVERRIDE=9.1.0 \
  external/local_config_cc/cc_wrapper.sh -fobjc-link-runtime -Wl,-S -shared -o bazel-out/darwin-fastbuild/bin/libgrpc.so bazel-out/darwin-fastbuild/bin/_objs/grpc/src/core/lib/surface/init.o bazel-out/darwin-fastbuild/bin/_objs/grpc/src/core/plugin_registry/grpc_plugin_registry.o -pthread -headerpad_max_install_names -lc++ -no-canonical-prefixes -undefined dynamic_lookup)

Use --sandbox_debug to see verbose messages from the sandbox
clang: warning: argument unused during compilation: '-pthread' [-Wunused-command-line-argument]
ld: illegal thread local variable reference to regular symbol __ZN9grpc_core7ExecCtx9exec_ctx_E for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
Target //:grpc failed to build

There are certainly differences but nothing seems to speak to thread-local variables as far as I can tell.

[vjpai]

We've had some further discussion about this in grpc/grpc#13856 (cc @yashykt) . I can make this work by using thread-local variables based on pthread_key functions rather than using the __thread declaration. This seems like what we would expect from a versioning issue. Is there a way to tell what version of clang is being used with bazel or any other versioning info that would help here?

[hlopko]

For osx-with-xcode toolchain it's a little bit complicated, you have to take a look into generated external/local_config_cc/cc_wrapper.sh file, you'll see the compiler invocation there.

[vjpai]

Hmm, found that and it's just an invocation of /usr/bin/clang which is the same as my standard c++

$ /usr/bin/clang -v
Apple LLVM version 9.0.0 (clang-900.0.38)
Target: x86_64-apple-darwin16.7.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin

[vjpai]

I have a workaround, if that helps. We have been using thread-local for quite a while in gRPC (using our own portable macros that effectively expand to __thread on Mac) but this was our first use of that in a header file. grpc/grpc#13916 removes the use of that in a header file and keeps it exclusively in the matching source file, and bazel builds with that just fine on Mac.

We can use this workaround for now, but I hope that we get a longer-term solution that allows us to use thread-local in header files at some point. I know that we're not the only ones to see this issue when building with bazel on mac.

[jtattermusch]

Looks like this is still a problem with bazel 1.0

[edbaunton]

I've just run across this problem using grpc 1.25.0 on Mac. If you clone https://gitlab.com/BuildGrid/buildbox/buildbox-common/blob/edbaunton/bazel/WORKSPACE and run bazel build //... the problem can be reproduced:

ld: illegal thread local variable reference to regular symbol __ZN9grpc_core7ExecCtx9exec_ctx_E for architecture x86_64

[jtattermusch]

@edbaunton your problem might have been solved by grpc/grpc#20510 (available in 1.26.0 release).

[NathanHowell]

@jtattermusch it's (still) failing for me with 1.26 as well as master:

$ bazel build :all
INFO: Running bazel wrapper (see //tools/bazel for details), bazel version 1.0.0 will be used instead of system-wide bazel installation.
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   615    0   615    0     0   2123      0 --:--:-- --:--:-- --:--:--  2120
100 39.2M  100 39.2M    0     0  1896k      0  0:00:21  0:00:21 --:--:-- 1302k
Starting local Bazel server and connecting to it...
INFO: Writing tracer profile to '/private/var/tmp/_bazel_nathan/029e4463ebaf163bdbac3eb656d38193/command.profile.gz'
INFO: SHA256 (https://boringssl.googlesource.com/boringssl/+archive/83da28a68f32023fd3b95a8ae94991a07b1f6c62.tar.gz) = 7b4fafe3e4af9d2acb33dfe18f22cc3b07d23bac3a53d3096d33ec0427268883
DEBUG: Rule 'boringssl' indicated that a canonical reproducible form can be obtained by modifying arguments sha256 = "7b4fafe3e4af9d2acb33dfe18f22cc3b07d23bac3a53d3096d33ec0427268883"
DEBUG: Call stack for the definition of repository 'boringssl' which is a http_archive (rule definition at /private/var/tmp/_bazel_nathan/029e4463ebaf163bdbac3eb656d38193/external/bazel_tools/tools/build_defs/repo/http.bzl:262:16):
 - /private/var/tmp/_bazel_nathan/029e4463ebaf163bdbac3eb656d38193/external/com_github_grpc_grpc/bazel/grpc_deps.bzl:125:9
 - /source/grpc/grpc/bazel/test/python_test_repo/WORKSPACE:8:1
INFO: Analyzed 19 targets (63 packages loaded, 3530 targets configured).
INFO: Found 19 targets...
ERROR: /private/var/tmp/_bazel_nathan/029e4463ebaf163bdbac3eb656d38193/external/com_github_grpc_grpc/BUILD:329:1: Linking of rule '@com_github_grpc_grpc//:grpc' failed (Exit 1) cc_wrapper.sh failed: error executing command external/local_config_cc/cc_wrapper.sh -lc++ -fobjc-link-runtime -Wl,-S -shared -o bazel-out/darwin-fastbuild/bin/external/com_github_grpc_grpc/libgrpc.so ... (remaining 8 argument(s) skipped)

Use --sandbox_debug to see verbose messages from the sandbox
ld: illegal thread local variable reference to regular symbol __ZN9grpc_core7ExecCtx9exec_ctx_E for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
INFO: Elapsed time: 78.994s, Critical Path: 21.15s
INFO: 200 processes: 200 darwin-sandbox.
FAILED: Build did NOT complete successfully
$ clang --version
Apple clang version 11.0.0 (clang-1100.0.33.16)
Target: x86_64-apple-darwin18.7.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin
$ git rev-parse HEAD
528dbb13e400654f9c73d8e26e70be48f27052c4
$ bazel version
Bazelisk version: v1.2.1
INFO: Running bazel wrapper (see //tools/bazel for details), bazel version 1.0.0 will be used instead of system-wide bazel installation.
Build label: 1.0.0
Build target: bazel-out/darwin-opt/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar
Build time: Thu Oct 10 10:19:27 2019 (1570702767)
Build timestamp: 1570702767
Build timestamp as int: 1570702767

[jtattermusch]

Ok, the fix is actually in grpc/grpc#13929 (which has been merged a long time ago).

For me, the build does pass (bazel 1.0.0 and clang Apple LLVM version 9.1.0)
bazel build :all -> INFO: Build completed successfully, 1361 total actions

Can you double check that your build actually defines GPR_PTHREAD_TLS (and not GPR_GCC_TLS) as seen here: https://github.com/grpc/grpc/pull/13929/files#diff-dd49c38c34f9401e10f1921931d6c121R215 . For that to happen, you need to have build --copt -DGRPC_BAZEL_BUILD in your bazelrc
https://github.com/grpc/grpc/pull/13929/files#diff-4c3a4e61471476647ad1c0987a744414R2

[NathanHowell]

@jtattermusch yeah, that works but now this configuration option leaks into my project - i'm trying to use the grpc rules and not just build grpc. is there some way to embed this into the grpc workspace?

[edbaunton]

The #define is checked in a header, so presumably anyone who includes the header would also need to set the requisite define. I am not a bazel expert but for a gRPC user to be able to get this flag transparently, the gRPC rules might be able to leverage this new Skylark functionality this.

Should this flag be enabled everywhere or only on Mac?

[jtattermusch]

@NathanHowell I'm happy to accept patches (agreed that specifying the --copt in the .rc file might not be ideal).

[jtattermusch]

Looks like defining --copt=-DGRPC_BAZEL_BUILD in our bazel.rc file is not ideal as the users don't inherit this setting (and they end up with wrong type of thread locals being used on mac).

[trybka]

With the current shipping version(s) of Xcode, it is impossible.

While there are some tests that will happen to work, it's not a mode that Bazel can/should support on Darwin.

Given that -undefined dynamic_lookup is deprecated, it seems unlikely that Apple will fix this.

[jbms]

I think -undefined dynamic_lookup is also how Python extension modules are built on macOS, unless the information at this link is outdated:
boostorg/build#69

Separately, though, what prevents bazel from providing the shared library dependencies to the linker?

[Wyverald]

@trybka It looks like your fix (ec55533) is causing breakages in rules_go and rules_rust on macOS. Could you take a look?

[jyknight]

rules_rust

At first glance I'd've said that the BUILD file is simply broken: https://github.com/bazelbuild/rules_rust/blob/8c388e1b816d0a7e5a7d3cc5d213be7f35299cf5/examples/ffi/rust_calling_c/c/BUILD#L23

A cc_library named "native_matrix_so" implicitly generates an output named "libnative_matrix_so.so" (along with "libnative_natrix_so.a"), yet, this build file declares a separate cc_binary rule which has the same name. Thus, the output name conflicts. So, the failure looks expected. Only... apparently it was working before. That it was working seems unexpected to me!

Ahh...so, here's the Bazel code which registers the failure outputs:

bazel/src/main/java/com/google/devtools/build/lib/rules/cpp/CcLibrary.java

Line 254 in 308bce3

if (!createDynamicLibrary && !supportsDynamicLinker) {

That code generates a dummy "error" output, when !supportsDynamicLinker. But, in the case where it is supported, it only generates a lib*.so output when srcs is not empty. So...that explains things here.

This is unfortunate -- and perhaps could be improved in Bazel, so that the behavior is more consistent. But, this also seems like a bug in the example BUILD file to have written rules with conflicting (or potentially-conflicting) output names, and can be easily solved by renaming one of those two rules.

rules_go

This is a different issue. Looking at the rules_go build file:
https://github.com/bazelbuild/rules_go/blob/5e733237761fbe70d10afd3156e73355392a66b2/tests/legacy/examples/cgo/cc_dependency/BUILD.bazel#L22

For some reason, this seems to go to extra effort to do something broken -- it's explicitly using the nodeps shared-library output from cc_library (via the filegroup asking for output_group = "dynamic_library",). This is odd, and I don't know any good reason why it'd want to do that. I imagine someone was just confused and wrote this accidentally?

The typical way to do this -- and what this example ought to be doing I think -- is to use a cc_binary instead of the cc_library/filegroup pair. Something like:

cc_binary(
    name = "c_version_so",
    srcs = ["c_version.c", "c_version.h"],
    linkshared=True,
)