tensorflow does not build on macOS

[essandess]

Click to expand!

Issue Type

Bug

Source

source

Tensorflow Version

2.11.0

Custom Code

No

OS Platform and Distribution

macOS 12.6.1 21G217 x86_64

Mobile device

No response

Python version

3.10, 3.9, 3.8

Bazel version

5.3.2

GCC/Compiler version

Xcode 14.1 14B47b

CUDA/cuDNN version

No response

GPU model and memory

No response

Current Behaviour?

I am trying to update the MacPorts distribution of tensorflow. It fails to build with the latest version 2.11.0. The build uses Xcode.

Related:

• py-tensorflow: Update to version 2.12.0, Add Python 310 macports/macports-ports#15397
• impossibility to build from source Mac M1 #57936

Standalone code to reproduce the issue

sudo port -pNc install py310-tensorflow

The MacPorts build recipe has worked up until the last version or so: https://github.com/macports/macports-ports/blob/master/python/py-tensorflow/Portfile

### Relevant log output

```shell
:debug:build system -W /opt/local/var/macports/build/_opt_local_ports_python_py-tensorflow/py310-tensorflow/work/tensorflow-2.11.0: PATH=/opt/local/var/macports/build/_opt_local_ports_python_py-tensorflow/py310-tensorflow/work/bazelwrap:/opt/local/libexec/bazel/bin:/opt/local/var/macports/build/_opt_local_ports_python_py-tensorflow/py310-tensorflow/work/bin:/opt/local/bin:/opt/local/sbin:/bin:/sbin:/usr/bin:/usr/sbin BAZEL_SH=/bin/bash CC_OPT_FLAGS=-march=x86-64 /opt/local/libexec/bazel/bin/bazel --max_idle_secs=10 --output_user_root=/opt/local/var/macports/build/_opt_local_ports_python_py-tensorflow/py310-tensorflow/work/bazel_build  fetch //tensorflow/tools/pip_package:build_pip_package

…

:info:build ld: malformed trie, childNodeOffset==0 file 'bazel-out/darwin-opt/bin/_solib_darwin_x86_64/libtensorflow_Spython_S_Upywrap_Utensorflow_Uinternal.so'
:info:build clang: error: linker command failed with exit code 1 (use -v to see invocation)
:info:build Error in child process '/usr/bin/xcrun'. 1
:info:build external/local_config_cc/cc_wrapper.sh: line 69: 75369 Abort trap: 6           "$(/usr/bin/dirname "$0")"/wrapped_clang "$@"
:info:build Target //tensorflow/tools/pip_package:build_pip_package failed to build
:info:build INFO: Elapsed time: 7166.337s, Critical Path: 421.62s

[sushreebarsa]

@essandess Could you make sure to follow the instructions mentioned here and check the tested build configurations .
Thank you!

[essandess]

@essandess Could you make sure to follow the instructions mentioned here and check the tested build configurations .
Thank you!

That’s exactly what I have done. The failed build command provided above is exactly that specified in your link. It worked with previous versions of tensorflow. It no longer works. There are multiple independent reports online and issues reporting this problem, e.g. #57936.

I’m simply trying to update the MacPorts Portfile that builds from source, and this fails at the end in with a linker failure, noted above and the other issues reported online.

[freedomtan]

@essandess FYR, as I commented before in #58368 (comment), it's likely a linked (ld) or linker+compiler problem. I replaced ld in Xcode 14.x with the one from Xcode 13.x, then I can successfully build it.

[essandess]

@essandess FYR, as I commented before in #58368 (comment), it's likely a linked (ld) or linker+compiler problem. I replaced ld in Xcode 14.x with the one from Xcode 13.x, then I can successfully build it.

Thanks. Generally there's only the latest available Xcode available on macOS, but there is ld64's ld linker. In fact, this ld64 binary is the one found in my own build attempt via $PATH:

$ which ld
/opt/local/bin/ld
$ port provides `which ld`
/opt/local/bin/ld is provided by: ld64

Clearly, bazel is hardcoding a different linker from Xcode somehow. How does one coerce bazel to use a specific linker?

[essandess]

I tried setting bazel build --linkopt="-fuse-ld=${prefix}/bin/ld" but this also fails with the same error.

[freedomtan]

I don't think bazel uses a different linker. Yes, on my mac, I have /opt/local/bin/ld installed via macport, too. But, that's symbolic links and shell script wrappers to Xcode ld.

$ ls -l /opt/local/bin/ld
lrwxr-xr-x  1 root  admin  8 Nov 22 12:03 /opt/local/bin/ld -> ld-xcode
myway:tmp freedom$ ls -l /opt/local/bin/ld-xcode 
lrwxr-xr-x  1 root  admin  24 Oct 27 13:58 /opt/local/bin/ld-xcode -> ../libexec/ld64/ld-xcode
$ ls -l /opt/local/libexec/ld64/ld-xcode
-rwxr-xr-x  1 root  admin  169 Oct 27 13:58 /opt/local/libexec/ld64/ld-xcode
$ cat  /opt/local/libexec/ld64/ld-xcode
#!/bin/bash

if [[ -x /usr/bin/xcrun ]] ; then
    exec /usr/bin/xcrun ld "${@}"
elif [[ -x /usr/bin/ld ]] ; then
    exec /usr/bin/ld "${@}"
else
    exec ld "${@}"
fi

And

$ xcrun -f ld 
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ld

That why I replaced the ld in /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ as I commented in #58368 (comment).

You can find older ld binaries in Xcode 13.x or Command Line Tools 13.x from Apple Developer site.

[murphycrosby]

@freedomtan is exactly right. I went ahead and documented the steps I followed here Building Tensorflow and Tensorflow Text on a M1 Mac

[essandess]

@freedomtan is exactly right. I went ahead and documented the steps I followed here Building Tensorflow and Tensorflow Text on a M1 Mac

That's awesome and everything for hacking one's own personal TF binary. But if one is trying to accomplish what I'm trying to accomplish: a general build formula that can be used to provide TF binaries across a suite of platforms in the MacPorts package manager, then not so much.

Much better if this upstream would test the releases against up-to-date build configurations and avoid this whole reporting and hacking cycle.

[sachinprasadhs]

Hi, Your issue seems similar to the issue here #58368, could you please close this issue and track the progress in that thread, since it is already involving lots of discussion.
You can follow the instructions here for the M1 mac installations of Tensorflow prebuilt library here https://developer.apple.com/metal/tensorflow-plugin/

[essandess]

You can follow the instructions here for the M1 mac installations of Tensorflow prebuilt library here https://developer.apple.com/metal/tensorflow-plugin/

That’s prebuilt for specific python versions, and doesn’t cover all of them.

This issue is about TF failing to compile from source on macOS.

[google-ml-butler]

This issue has been automatically marked as stale because it has no recent activity. It will be closed if no further activity occurs. Thank you.