gen_wkt_cc_sources failed: dyld[69712]: missing symbol called on Mac ARM64

[phallot]

What version of protobuf and what language are you using?
Version: v22.1 (also happens with 22.0, but not before)
Language: C++

What operating system (Linux, Windows, ...) and version?
MacOs Version 13.2.1 (22D68) ARM64

What runtime / compiler are you using (e.g., python version or gcc version)
Homebrew clang version 15.0.7
Bazel 6.1
Python 3.11.2

What did you do?
Steps to reproduce the behavior:

1. Followed the instructions from https://github.com/protocolbuffers/protobuf/blob/main/src/README.md to get the sources using git
2. bazel build :protoc :protobuf

What did you expect to see
Successful build

What did you see instead?

ERROR: /Users/foo/dev/tmp/protobuf/src/google/protobuf/BUILD.bazel:117:8: Executing genrule //src/google/protobuf:gen_wkt_cc_sources failed: (Aborted): bash failed: error executing command (from target //src/google/protobuf:gen_wkt_cc_sources) /bin/bash -c ... (remaining 1 argument skipped)

Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging
dyld[69712]: missing symbol called
/bin/bash: line 1: 69712 Abort trap: 6           bazel-out/darwin_arm64-opt-exec-2B5CBBC6/bin/src/google/protobuf/compiler/protoc_nowkt --cpp_out=dllexport_decl=PROTOBUF_EXPORT:bazel-out/darwin_arm64-fastbuild/bin/src/google/protobuf/wkt --proto_path=$(dirname $(dirname $(dirname src/google/protobuf/any.proto))) src/google/protobuf/any.proto src/google/protobuf/api.proto src/google/protobuf/duration.proto src/google/protobuf/empty.proto src/google/protobuf/field_mask.proto src/google/protobuf/source_context.proto src/google/protobuf/struct.proto src/google/protobuf/timestamp.proto src/google/protobuf/type.proto src/google/protobuf/wrappers.proto
ERROR: /Users/foo.hallot/dev/tmp/protobuf/src/google/protobuf/BUILD.bazel:142:11 Middleman _middlemen/_S_Ssrc_Sgoogle_Sprotobuf_Cwkt_Ucc_Uproto-cc_library-compile failed: (Aborted): bash failed: error executing command (from target //src/google/protobuf:gen_wkt_cc_sources) /bin/bash -c ... (remaining 1 argument skipped)

There are also repeated warnings like:
warning: unknown warning option '-Wno-deprecated-non-prototype' [-Wunknown-warning-option]
Those are not build-breaking, but maybe they indicate a toolchain issue?

It also looks like I am not the only one having this issue:
https://stackoverflow.com/questions/75551678/bazel-build-proto-failed-due-to-missing-symbol-called-in-mac

[mkruskal-google]

Hi, thanks for reporting this!

I'm unable to reproduce this. I have an M1 macbook with macos 13.2.1, bazel 6.1, and clang 15.0.7 and I can build 22.1 just fine. If you modify .bazelrc to use --cxxopt=-std=c++20 instead of C++14 does it fix the problem? It looks like that's the fix for the stack overflow issue you pointed at.

Otherwise, could you run with --sandbox_debug -s and give us the logs for that?

[phallot]

Hi @mkruskal-google, thanks for the reply.

Unfortunately, changing to --cxxopt=-std=c++20 did not help.

It looks like that's the fix for the stack overflow issue you pointed at.

I don't think that's what's actually written. There are two errors mentioned.
Adding --cxxopt=-std=c++20 fixed the C++ versions less than C++14 are not supported. error, not the symbol error.
Which makes sense, since in the SO issue they are not compiling protobuf directly, so they're not using the .bazelrc of protobuf, but Bazel's default C++ standard (c++0x). The abseil dependency of protobuf is then the one complaining.

To go back to the original issue, I have attached the full logs with the debug options.
While playing with my toolchains, I have also noticed that gcc is called, which on my system defaults to Apple clang version 14.0.0 (clang-1400.0.29.202).
I am pretty new to Bazel, and I don't really understand how toolchains work, nor which toolchain is used by default.
Could it be that Bazel is attempting to compile thinking it is using gcc, but it is actually calling the apple Clang 14 compiler?
That would also explain the unknown warning option '-Wno-deprecated-non-prototype' warnings, which are new to clang 15.

If so, I am not exactly sure what's the best way to fix.

EDIT: Doing something like export CC=clang; export CXX=clang++, where clang is clang 15 from Homebrew fixes the deprecated-non-prototype warnings, but not the symbol issue.

[sluongng]

We ran into a similar issue on our end:

Executing genrule @com_google_protobuf//src/google/protobuf:gen_wkt_cc_sources
...
dyld[97646]: symbol not found in flat namespace '_CFRelease'

This was caused by bazel-out/darwin_arm64-opt-exec-2B5CBBC6/bin/external/com_google_protobuf/src/google/protobuf/compiler/protoc_nowkt was linked without the CoreFoundation framework in Mac. Running this binary alone without any flag / args would still result in that error.

This is a small patch that would fix it

--- /tmp/cpp_opts.bzl.original  2023-03-22 11:17:55.231353503 +0700
+++ bazel-buildbuddy/external/com_google_protobuf/build_defs/cpp_opts.bzl       2023-03-22 11:26:26.558827766 +0700
@@ -36,6 +36,11 @@
         # Suppress linker warnings about files with no symbols defined.
         "-ignore:4221",
     ],
+    "@platforms//os:macos": [
+        "-lpthread",
+        "-lm",
+        "-framework CoreFoundation",
+    ],
     "//conditions:default": [
         "-lpthread",
         "-lm",

[sluongng]

Before the patch

> otool -dyld_info bazel-out/darwin_arm64-opt-exec-2B5CBBC6/bin/external/com_google_protobuf/src/google/protobuf/compiler/protoc_nowkt | grep '_CF'
__DATA_CONST __got         0x10035C038 0x8010000000000007 bind   0x0      flat-namespace _CFRelease
__DATA_CONST __got         0x10035C040 0x8010000000000008 bind   0x0      flat-namespace _CFStringGetCString
__DATA_CONST __got         0x10035C048 0x8010000000000009 bind   0x0      flat-namespace _CFStringGetLength
__DATA_CONST __got         0x10035C050 0x801000000000000A bind   0x0      flat-namespace _CFStringGetMaximumSizeForEncoding
__DATA_CONST __got         0x10035C058 0x801000000000000B bind   0x0      flat-namespace _CFTimeZoneCopyDefault
__DATA_CONST __got         0x10035C060 0x801000000000000C bind   0x0      flat-namespace _CFTimeZoneGetName

After the patch

> otool -dyld_info bazel-out/darwin_arm64-opt-exec-2B5CBBC6/bin/external/com_google_protobuf/src/google/protobuf/compiler/protoc_nowkt | grep '_CF'
__DATA_CONST __got         0x10035C038 0x8010000000000007 bind   0x0      CoreFoundation _CFRelease
__DATA_CONST __got         0x10035C040 0x8010000000000008 bind   0x0      CoreFoundation _CFStringGetCString
__DATA_CONST __got         0x10035C048 0x8010000000000009 bind   0x0      CoreFoundation _CFStringGetLength
__DATA_CONST __got         0x10035C050 0x801000000000000A bind   0x0      CoreFoundation _CFStringGetMaximumSizeForEncoding
__DATA_CONST __got         0x10035C058 0x801000000000000B bind   0x0      CoreFoundation _CFTimeZoneCopyDefault
__DATA_CONST __got         0x10035C060 0x801000000000000C bind   0x0      CoreFoundation _CFTimeZoneGetName

[Chellapandi]

Hi, I am getting below error. Any fix or work around for this?

external/com_google_protobuf/src/google/protobuf/BUILD.bazel:117:8: Executing genrule @com_google_protobuf//src/google/protobuf:gen_wkt_cc_sources failed: (Aborted): bash failed: error executing command /bin/bash -c ... (remaining 1 argument skipped)

Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging
dyld[10570]: missing symbol called
/bin/bash: line 1: 10570 Abort trap: 6           bazel-out/darwin-opt-exec-2B5CBBC6/bin/external/com_google_protobuf/src/google/protobuf/compiler/protoc_nowkt --cpp_out=dllexport_decl=PROTOBUF_EXPORT:bazel-out/darwin-opt-exec-2B5CBBC6/bin/external/com_google_protobuf/src/google/protobuf/wkt --proto_path=$(dirname $(dirname $(dirname external/com_google_protobuf/src/google/protobuf/any.proto))) external/com_google_protobuf/src/google/protobuf/any.proto external/com_google_protobuf/src/google/protobuf/api.proto external/com_google_protobuf/src/google/protobuf/duration.proto external/com_google_protobuf/src/google/protobuf/empty.proto external/com_google_protobuf/src/google/protobuf/field_mask.proto external/com_google_protobuf/src/google/protobuf/source_context.proto external/com_google_protobuf/src/google/protobuf/struct.proto external/com_google_protobuf/src/google/protobuf/timestamp.proto external/com_google_protobuf/src/google/protobuf/type.proto external/com_google_protobuf/src/google/protobuf/wrappers.proto

[Mizux]

@Chellapandi seems your message is trunked after the missing symbol called ????

otherwise you'll need: https://github.com/protocolbuffers/protobuf/pull/12320/files

Personally I'm using:
https://github.com/Mizux/bazel-proto/blob/6befadbd764e081a7068229f9c7bceccaf1997fe/patches/protobuf-v22.2.patch
and
https://github.com/Mizux/bazel-proto/blob/6befadbd764e081a7068229f9c7bceccaf1997fe/WORKSPACE#L63-L68

But there is still plenty of errors due to bad management of copt by bazel/protobuf/rules_cc (still investigating...) and poor (i.e. no) support of macos m1 in kokoro or github workflow...

[Chellapandi]

@Chellapandi seems your message is trunked after the missing symbol called ????

otherwise you'll need: https://github.com/protocolbuffers/protobuf/pull/12320/files

Personally I'm using: https://github.com/Mizux/bazel-proto/blob/main/patches/protobuf-v22.2.patch and https://github.com/Mizux/bazel-proto/blob/6befadbd764e081a7068229f9c7bceccaf1997fe/WORKSPACE#L63-L68

But there is still plenty of errors due to bad management of copt by bazel/protobuf/rules_cc (still investigating...) and poor (i.e. no) support of macos m1 in kokoro or github workflow...

@Mizux Thanks for your quick response. missing symbol error has been fixed by the fix that you have provided. But still I am getting below error.

v1/service.pb.gw.go:16:2: imported and not used: "github.com/golang/protobuf/ptypes/empty"
v1/service.pb.gw.go:36:15: undefined: emptypb
v1/service.pb.gw.go:53:15: undefined: emptypb

I have raised an issue here. Please let me know if you faced the same issue.