build on macOS 11 (beta) does not find system-supplied third-party libraries

[ned-deily]

BPO	41116
Nosy	@ronaldoussoren, @ned-deily, @ambv, @miss-islington, @sjmulder, @dtrodrigues
PRs	bpo-41116: Ensure system supplied libraries are found on macOS 11 #23301 [3.9] bpo-41116: Ensure system supplied libraries are found on macOS 11 (GH-23301) #23455 bpo-41116: Fix setup.py test for macOS Tcl/Tk frameworks #23649 [3.9] bpo-41116: Fix setup.py test for macOS Tcl/Tk frameworks (GH-23649) #23650

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2021-05-02.09:38:07.772>
created_at = <Date 2020-06-25.15:51:04.461>
labels = ['OS-mac', '3.8', '3.9', '3.10']
title = 'build on macOS 11 (beta) does not find system-supplied third-party libraries'
updated_at = <Date 2021-05-02.09:38:07.772>
user = 'https://github.com/ned-deily'

bugs.python.org fields:

activity = <Date 2021-05-02.09:38:07.772>
actor = 'ned.deily'
assignee = 'none'
closed = True
closed_date = <Date 2021-05-02.09:38:07.772>
closer = 'ned.deily'
components = ['macOS']
creation = <Date 2020-06-25.15:51:04.461>
creator = 'ned.deily'
dependencies = []
files = []
hgrepos = []
issue_num = 41116
keywords = ['patch']
message_count = 19.0
messages = ['372379', '372415', '372416', '372417', '380556', '380811', '381589', '381598', '382553', '382554', '384679', '384680', '384846', '384884', '384961', '386895', '386901', '386902', '392679']
nosy_count = 8.0
nosy_names = ['ronaldoussoren', 'ned.deily', 'lukasz.langa', 'miss-islington', 'sjmulder', 'dtrodrigues', 'seb', 'richiejthomas']
pr_nums = ['23301', '23455', '23649', '23650']
priority = 'high'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue41116'
versions = ['Python 3.8', 'Python 3.9', 'Python 3.10']

[ned-deily]

When building on macOS 11 (beta), a number of modules that should normally build on macOS fail because the system-supplied third-party libraries are not found.

The necessary bits to build these optional modules were not found:
_bz2 _curses _curses_panel
_gdbm _hashlib _lzma
_ssl ossaudiodev readline
spwd zlib

The list should look like this (with no additional third-party libs supplied from another source like Homebrew or MacPorts):

The necessary bits to build these optional modules were not found:
_gdbm _hashlib _ssl
ossaudiodev spwd

The problem is due to a change in the 11 beta versus 10.15 or earlier systems:
"New in macOS Big Sur 11 beta, the system ships with a built-in dynamic linker cache of all system-provided libraries. As part of this change, copies of dynamic libraries are no longer present on the filesystem. Code that attempts to check for dynamic library presence by looking for a file at a path or enumerating a directory will fail. Instead, check for library presence by attempting to dlopen() the path, which will correctly check for the library in the cache."

This breaks tests in setup.py using find_library_file() to determine if a library is present and in what directory it exists. setup.py depends on Lib/distutils/unixccompiler.py to do the dirty work. A similar problem arose on earlier macOS releases when header files could no longer be installed in the systems /usr/include; setup.py had to be taught to look in the SDK being used implicitly or explicitly by the compiler preprocessor. We could probably do something like that here while trying to avoid changes that might break downstream supplements/replacements to distutils, for example, setuptools.

There is a workaround: explicitly specify the SDK location to ./configure (you also need to specify the universal archs setting):

./configure \
--enable-universalsdk=$(xcodebuild -version -sdk macosx Path) \
--with-universal-archs=intel-64 \
...

[ronaldoussoren]

These build for me (master branch), because the support for .tbd files in the macOS SDK still works.

The ones that don't build for me:

• Bits not found: _gdbm, _hashlib, _ssl, ossaudiodev, spwd
• Failed to build: _ctypes, _decimal, _lzma

This was with a "Universal 2" build (which explains _ctypes) and without 3party libraries (which explains _gdbm, _ssl and _lzma). ossaudiodev and spwd are not supported on macOS even on older versions of the OS.

I can get _decimal to build with a trivial patch to Modules/_decimal/mpdecimal.py (I'll create a PR later today). The PR will obviously be untested w.r.t. actually supporting ARM64.

[ned-deily]

I should have specified that the failures I saw were just with the default intel-64 arch on the beta.

After thinking about it a bit and reviewing the code, I think the best approach is to handle this in _osx_support.py as part of the compiler flags customization. What's been happening over the years is that Apple has been deprecating and then removing system header files and system library files on running systems and instead forcing the use of an SDK to build with for both. That's fine but, to continue to support a range of systems, we should be able to handle the various default configurations when the user does not explicitly specify a particular SDK to use. The Apple compiler chain does the right thing under the covers so that C compilations just work. setip.py's macosx_sdk_root() now has initialization code to discover the path to the system header files that the compiler is using; it calls the compiler with -v on a dummy program and then scans the compiler output for the include paths searched until it finds either /usr/include or a path ending in .sdk/usr/include. (It's a bit of a hack but a fairly widely-used hack that works with both clang and gcc, at least, and for the Apple-supplied compilers, takes into account the dynamic specification of SDK via xcrun.)

I think the thing to do is move that sdk path detection into _osx_support and then, at the end of other customization steps, if there isn't already a valid -isysroot provided by the user (either through ./configure or CFLAGS et al, add a -isysroot for the discovered default SDK path (or /) into the appropriate customized compiler and linker flags. It would also provide a function to return the SDK path for setup.py (the equivalent of the current macosx_sdk_root()) and others to use. setup.py and Distutils should then do the right things for both header and library file paths on any configuration (whether header files are installed or not and whether lib files are installed or not, Command Line Tools or Xcode) and, with luck, any other Distutils replacements that also use _osx_support will also do the right thing.

I had contemplated removing _osx_support at some point in the future but, now with the return of universal files, I'm glad we didn't.

[ronaldoussoren]

I see, and (as expected) get the same output when I build like that.

I agree that we should teach _osx_support about this. I guess we should add a function that returns the SDK root, which can then be used in distutils (and setuptools, ...) instead of looking for -isysroot in CFLAGS. The new function can than look for the default SDK when there is no sysroot flag.

For 3.10 we should look into cleaning up configure.ac, it contains some logic that is no longer valid or necessary (such as looking for the architectures supported by /usr/lib/libSystem.dylib). We should also avoid adding "-isysroot" to the CFLAGS/LDFLAGS to avoid problems when building on user machines that might have a different version of Xcode.

[ronaldoussoren]

So close...

the problem we're running into is that "-isysroot" is only added for universal builds, not for regular builds. Furthermore unixccompiler doesn't know that is should always look in the SDK and not in the regular location (for system locations).

I guess we should switch to:

• Never add -isysroot to CFLAGS (change to configure)
• Teach _osx_support about locating the default sdk root, both for Xcode and "command line tools" (_osx_support.default_sdk_root())
• Use that new function in setup.py and unixccompiler.

The first item might affect older systems where Xcode shipped with 2 SDKs (current and previous OS version).

Is that something we can back port to 3.8 and 3.9?

A less clean, but smaller, change is to teach the configure script to always add -isysroot on macOS, not just when doing a universal build.

[sjmulder]

With './configure; make' of master on macOS 11 on Apple Silicon I get the library detection failure. The quick and easy fix was to amend inc_dirs and lib_dirs in setup.py:

         if MACOS:
+            sysroot = macosx_sdk_root()
+            self.inc_dirs.append(sysroot + '/usr/include')
+            self.lib_dirs.append(sysroot + '/usr/lib')

Interestingly the lzma library is found, but the module fails because it cannot find lzma.h, which does indeed appear to be missing in the SDK.

[ned-deily]

New changeset 404a719 by Ronald Oussoren in branch 'master':
bpo-41116: Ensure system supplied libraries are found on macOS 11 (GH-23301)
404a719

[ned-deily]

New changeset 0aab352 by Miss Islington (bot) in branch '3.9':
bpo-41116: Ensure system supplied libraries are found on macOS 11 (GH-23301) (GH-23455)
0aab352

[ned-deily]

New changeset 29afab6 by Ned Deily in branch 'master':
bpo-41116: Fix setup.py test for macOS Tcl/Tk frameworks (GH-23649)
29afab6

[ned-deily]

New changeset 06002b3 by Miss Islington (bot) in branch '3.9':
[3.9] bpo-41116: Fix setup.py test for macOS Tcl/Tk frameworks (GH-23649) (GH-23650)
06002b3

[seb]

I am running macOS Big Sur Version 11.1 on Silicon and still get the error about the missing lzma.h file. I can confirm that I use the latest Python 3.9.1 version which includes the patches of this issue.

gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Wstrict-prototypes -Werror=implicit-function-declaration -fvisibility=hidden -I./Include/internal -I./Include -I. -I/usr/local/include -I/Users/dev/Downloads/Python-3.9.1/Include -I/Users/dev/Downloads/Python-3.9.1 -c /Users/dev/Downloads/Python-3.9.1/Modules/_lzmamodule.c -o build/temp.macosx-11.1-arm64-3.9/Users/dev/Downloads/Python-3.9.1/Modules/_lzmamodule.o
/Users/dev/Downloads/Python-3.9.1/Modules/_lzmamodule.c:16:10: fatal error: 'lzma.h' file not found
#include <lzma.h>

Is this supposed to be still the case? Thank you!

[ned-deily]

I am running macOS Big Sur Version 11.1 on Silicon and still get the error about the missing lzma.h file.

Unfortunately, Apple does not include the XZ library with macOS so, to build that module, you will need to supply a copy from another source, either build it yourself or from a third-party library provider like Homebrew or MacPorts. The Python Developer's Guid has some suggestions:

https://devguide.python.org/setup/#macos-and-os-x

[seb]

Thanks for the help! I tried the instructions, without success. I installed xz through homebrew (which needs to be installed on Silicon under /opt/homebrew). I can confirm the existance of: /opt/homebrew/Cellar/xz/5.2.5/include/lzma.h

I used CPPFLAGS and also modified the system_include_dirs variable in the setup.py file, in both cases lzma.h seems to be ignored.

system_include_dirs = ['opt/homebrew/include', '/opt/homebrew/Cellar/xz/5.2.5/include']

Am I missing something obvious here? Thanks!

[dtrodrigues]

If your 'brew --prefix' is /opt/homebrew, then setting

CFLAGS="-I/opt/homebrew/include" CPPFLAGS="-I/opt/homebrew/include" LDFLAGS="-L/opt/homebrew/lib"

when running ./configure appears to be sufficient to build the interpreter with lzma support once you've already done 'brew install xz'.

[seb]

I tried that before, also with your command, without success. I saw that the CFLAGS includes weren't forwarded to the compiler - I don't know if that is only an issue here on my system. So instead I modified the setup.py and added the directory manually. It now works. Thanks for your help!!

[richiejthomas]

Hello, can you give me a few more details around the workaround for this. I am using pyenv to install 3.9.1 and not sure how I would include the xz installed using homebrew. Where do I find the setup.py or configure command? Thanks for the help!

[dtrodrigues]

You'll probably need to contact the pyenv project (https://github.com/pyenv/pyenv) if the following doesn't work for you. However, doing something like

CFLAGS="-I$(brew --prefix)/include" CPPFLAGS="-I$(brew --prefix)/include" LDFLAGS="-L$(brew --prefix)/lib" pyenv install 3.9.1

is working for me to use the lzma module if you already have xz installed with Homebrew.

[richiejthomas]

That works for me. Thank you!

On Fri, Feb 12, 2021 at 11:12 PM Dustin Rodrigues <report@bugs.python.org>
wrote:

Dustin Rodrigues <dust.rod@gmail.com> added the comment:

You'll probably need to contact the pyenv project (
https://github.com/pyenv/pyenv) if the following doesn't work for you.
However, doing something like

CFLAGS="-I$(brew --prefix)/include" CPPFLAGS="-I$(brew --prefix)/include"
LDFLAGS="-L$(brew --prefix)/lib" pyenv install 3.9.1

is working for me to use the lzma module if you already have xz installed
with Homebrew.

----------

---

Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue41116\>

---

[ambv]

New changeset b29d0a5 by Ned Deily in branch '3.8':
[3.8] bpo-41100: Support macOS 11 Big Sur and Apple Silicon Macs (bpo-25806)
b29d0a5