Skip to content

build on macOS 11 (beta) does not find system-supplied third-party libraries #85288

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
ned-deily opened this issue Jun 25, 2020 · 19 comments
Closed
Labels
3.8 (EOL) end of life 3.9 only security fixes 3.10 only security fixes OS-mac

Comments

@ned-deily
Copy link
Member

BPO 41116
Nosy @ronaldoussoren, @ned-deily, @ambv, @miss-islington, @sjmulder, @dtrodrigues
PRs
  • bpo-41116: Ensure system supplied libraries are found on macOS 11 #23301
  • [3.9] bpo-41116: Ensure system supplied libraries are found on macOS 11 (GH-23301) #23455
  • bpo-41116: Fix setup.py test for macOS Tcl/Tk frameworks #23649
  • [3.9] bpo-41116: Fix setup.py test for macOS Tcl/Tk frameworks (GH-23649) #23650
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
    closed_at = <Date 2021-05-02.09:38:07.772>
    created_at = <Date 2020-06-25.15:51:04.461>
    labels = ['OS-mac', '3.8', '3.9', '3.10']
    title = 'build on macOS 11 (beta) does not find system-supplied third-party libraries'
    updated_at = <Date 2021-05-02.09:38:07.772>
    user = 'https://github.com/ned-deily'

    bugs.python.org fields:

    activity = <Date 2021-05-02.09:38:07.772>
    actor = 'ned.deily'
    assignee = 'none'
    closed = True
    closed_date = <Date 2021-05-02.09:38:07.772>
    closer = 'ned.deily'
    components = ['macOS']
    creation = <Date 2020-06-25.15:51:04.461>
    creator = 'ned.deily'
    dependencies = []
    files = []
    hgrepos = []
    issue_num = 41116
    keywords = ['patch']
    message_count = 19.0
    messages = ['372379', '372415', '372416', '372417', '380556', '380811', '381589', '381598', '382553', '382554', '384679', '384680', '384846', '384884', '384961', '386895', '386901', '386902', '392679']
    nosy_count = 8.0
    nosy_names = ['ronaldoussoren', 'ned.deily', 'lukasz.langa', 'miss-islington', 'sjmulder', 'dtrodrigues', 'seb', 'richiejthomas']
    pr_nums = ['23301', '23455', '23649', '23650']
    priority = 'high'
    resolution = 'fixed'
    stage = 'resolved'
    status = 'closed'
    superseder = None
    type = None
    url = 'https://bugs.python.org/issue41116'
    versions = ['Python 3.8', 'Python 3.9', 'Python 3.10']

    @ned-deily
    Copy link
    Member Author

    When building on macOS 11 (beta), a number of modules that should normally build on macOS fail because the system-supplied third-party libraries are not found.

    The necessary bits to build these optional modules were not found:
    _bz2 _curses _curses_panel
    _gdbm _hashlib _lzma
    _ssl ossaudiodev readline
    spwd zlib

    The list should look like this (with no additional third-party libs supplied from another source like Homebrew or MacPorts):

    The necessary bits to build these optional modules were not found:
    _gdbm _hashlib _ssl
    ossaudiodev spwd

    The problem is due to a change in the 11 beta versus 10.15 or earlier systems:
    "New in macOS Big Sur 11 beta, the system ships with a built-in dynamic linker cache of all system-provided libraries. As part of this change, copies of dynamic libraries are no longer present on the filesystem. Code that attempts to check for dynamic library presence by looking for a file at a path or enumerating a directory will fail. Instead, check for library presence by attempting to dlopen() the path, which will correctly check for the library in the cache."

    This breaks tests in setup.py using find_library_file() to determine if a library is present and in what directory it exists. setup.py depends on Lib/distutils/unixccompiler.py to do the dirty work. A similar problem arose on earlier macOS releases when header files could no longer be installed in the systems /usr/include; setup.py had to be taught to look in the SDK being used implicitly or explicitly by the compiler preprocessor. We could probably do something like that here while trying to avoid changes that might break downstream supplements/replacements to distutils, for example, setuptools.

    There is a workaround: explicitly specify the SDK location to ./configure (you also need to specify the universal archs setting):

    ./configure \
    --enable-universalsdk=$(xcodebuild -version -sdk macosx Path) \
    --with-universal-archs=intel-64 \
    ...

    @ned-deily ned-deily added 3.7 (EOL) end of life 3.10 only security fixes 3.8 (EOL) end of life 3.9 only security fixes OS-mac labels Jun 25, 2020
    @ronaldoussoren
    Copy link
    Contributor

    These build for me (master branch), because the support for .tbd files in the macOS SDK still works.

    The ones that don't build for me:

    • Bits not found: _gdbm, _hashlib, _ssl, ossaudiodev, spwd
    • Failed to build: _ctypes, _decimal, _lzma

    This was with a "Universal 2" build (which explains _ctypes) and without 3party libraries (which explains _gdbm, _ssl and _lzma). ossaudiodev and spwd are not supported on macOS even on older versions of the OS.

    I can get _decimal to build with a trivial patch to Modules/_decimal/mpdecimal.py (I'll create a PR later today). The PR will obviously be untested w.r.t. actually supporting ARM64.

    @ned-deily
    Copy link
    Member Author

    I should have specified that the failures I saw were just with the default intel-64 arch on the beta.

    After thinking about it a bit and reviewing the code, I think the best approach is to handle this in _osx_support.py as part of the compiler flags customization. What's been happening over the years is that Apple has been deprecating and then removing system header files and system library files on running systems and instead forcing the use of an SDK to build with for both. That's fine but, to continue to support a range of systems, we should be able to handle the various default configurations when the user does not explicitly specify a particular SDK to use. The Apple compiler chain does the right thing under the covers so that C compilations just work. setip.py's macosx_sdk_root() now has initialization code to discover the path to the system header files that the compiler is using; it calls the compiler with -v on a dummy program and then scans the compiler output for the include paths searched until it finds either /usr/include or a path ending in .sdk/usr/include. (It's a bit of a hack but a fairly widely-used hack that works with both clang and gcc, at least, and for the Apple-supplied compilers, takes into account the dynamic specification of SDK via xcrun.)

    I think the thing to do is move that sdk path detection into _osx_support and then, at the end of other customization steps, if there isn't already a valid -isysroot provided by the user (either through ./configure or CFLAGS et al, add a -isysroot for the discovered default SDK path (or /) into the appropriate customized compiler and linker flags. It would also provide a function to return the SDK path for setup.py (the equivalent of the current macosx_sdk_root()) and others to use. setup.py and Distutils should then do the right things for both header and library file paths on any configuration (whether header files are installed or not and whether lib files are installed or not, Command Line Tools or Xcode) and, with luck, any other Distutils replacements that also use _osx_support will also do the right thing.

    I had contemplated removing _osx_support at some point in the future but, now with the return of universal files, I'm glad we didn't.

    @ronaldoussoren
    Copy link
    Contributor

    I see, and (as expected) get the same output when I build like that.

    I agree that we should teach _osx_support about this. I guess we should add a function that returns the SDK root, which can then be used in distutils (and setuptools, ...) instead of looking for -isysroot in CFLAGS. The new function can than look for the default SDK when there is no sysroot flag.

    For 3.10 we should look into cleaning up configure.ac, it contains some logic that is no longer valid or necessary (such as looking for the architectures supported by /usr/lib/libSystem.dylib). We should also avoid adding "-isysroot" to the CFLAGS/LDFLAGS to avoid problems when building on user machines that might have a different version of Xcode.

    @ronaldoussoren
    Copy link
    Contributor

    So close...

    the problem we're running into is that "-isysroot" is only added for universal builds, not for regular builds. Furthermore unixccompiler doesn't know that is should always look in the SDK and not in the regular location (for system locations).

    I guess we should switch to:

    • Never add -isysroot to CFLAGS (change to configure)
    • Teach _osx_support about locating the default sdk root, both for Xcode and "command line tools" (_osx_support.default_sdk_root())
    • Use that new function in setup.py and unixccompiler.

    The first item might affect older systems where Xcode shipped with 2 SDKs (current and previous OS version).

    Is that something we can back port to 3.8 and 3.9?

    A less clean, but smaller, change is to teach the configure script to always add -isysroot on macOS, not just when doing a universal build.

    @sjmulder
    Copy link
    Mannequin

    sjmulder mannequin commented Nov 12, 2020

    With './configure; make' of master on macOS 11 on Apple Silicon I get the library detection failure. The quick and easy fix was to amend inc_dirs and lib_dirs in setup.py:

             if MACOS:
    +            sysroot = macosx_sdk_root()
    +            self.inc_dirs.append(sysroot + '/usr/include')
    +            self.lib_dirs.append(sysroot + '/usr/lib')

    Interestingly the lzma library is found, but the module fails because it cannot find lzma.h, which does indeed appear to be missing in the SDK.

    @ned-deily
    Copy link
    Member Author

    New changeset 404a719 by Ronald Oussoren in branch 'master':
    bpo-41116: Ensure system supplied libraries are found on macOS 11 (GH-23301)
    404a719

    @ned-deily
    Copy link
    Member Author

    New changeset 0aab352 by Miss Islington (bot) in branch '3.9':
    bpo-41116: Ensure system supplied libraries are found on macOS 11 (GH-23301) (GH-23455)
    0aab352

    @ned-deily
    Copy link
    Member Author

    New changeset 29afab6 by Ned Deily in branch 'master':
    bpo-41116: Fix setup.py test for macOS Tcl/Tk frameworks (GH-23649)
    29afab6

    @ned-deily
    Copy link
    Member Author

    New changeset 06002b3 by Miss Islington (bot) in branch '3.9':
    [3.9] bpo-41116: Fix setup.py test for macOS Tcl/Tk frameworks (GH-23649) (GH-23650)
    06002b3

    @seb
    Copy link
    Mannequin

    seb mannequin commented Jan 8, 2021

    I am running macOS Big Sur Version 11.1 on Silicon and still get the error about the missing lzma.h file. I can confirm that I use the latest Python 3.9.1 version which includes the patches of this issue.

    gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Wstrict-prototypes -Werror=implicit-function-declaration -fvisibility=hidden -I./Include/internal -I./Include -I. -I/usr/local/include -I/Users/dev/Downloads/Python-3.9.1/Include -I/Users/dev/Downloads/Python-3.9.1 -c /Users/dev/Downloads/Python-3.9.1/Modules/_lzmamodule.c -o build/temp.macosx-11.1-arm64-3.9/Users/dev/Downloads/Python-3.9.1/Modules/_lzmamodule.o
    /Users/dev/Downloads/Python-3.9.1/Modules/_lzmamodule.c:16:10: fatal error: 'lzma.h' file not found
    #include <lzma.h>

    Is this supposed to be still the case? Thank you!

    @ned-deily
    Copy link
    Member Author

    I am running macOS Big Sur Version 11.1 on Silicon and still get the error about the missing lzma.h file.

    Unfortunately, Apple does not include the XZ library with macOS so, to build that module, you will need to supply a copy from another source, either build it yourself or from a third-party library provider like Homebrew or MacPorts. The Python Developer's Guid has some suggestions:

    https://devguide.python.org/setup/#macos-and-os-x

    @seb
    Copy link
    Mannequin

    seb mannequin commented Jan 11, 2021

    Thanks for the help! I tried the instructions, without success. I installed xz through homebrew (which needs to be installed on Silicon under /opt/homebrew). I can confirm the existance of: /opt/homebrew/Cellar/xz/5.2.5/include/lzma.h

    I used CPPFLAGS and also modified the system_include_dirs variable in the setup.py file, in both cases lzma.h seems to be ignored.

    system_include_dirs = ['opt/homebrew/include', '/opt/homebrew/Cellar/xz/5.2.5/include']

    Am I missing something obvious here? Thanks!

    @dtrodrigues
    Copy link
    Mannequin

    dtrodrigues mannequin commented Jan 12, 2021

    If your 'brew --prefix' is /opt/homebrew, then setting

    CFLAGS="-I/opt/homebrew/include" CPPFLAGS="-I/opt/homebrew/include" LDFLAGS="-L/opt/homebrew/lib"

    when running ./configure appears to be sufficient to build the interpreter with lzma support once you've already done 'brew install xz'.

    @seb
    Copy link
    Mannequin

    seb mannequin commented Jan 12, 2021

    I tried that before, also with your command, without success. I saw that the CFLAGS includes weren't forwarded to the compiler - I don't know if that is only an issue here on my system. So instead I modified the setup.py and added the directory manually. It now works. Thanks for your help!!

    @richiejthomas
    Copy link
    Mannequin

    richiejthomas mannequin commented Feb 13, 2021

    Hello, can you give me a few more details around the workaround for this. I am using pyenv to install 3.9.1 and not sure how I would include the xz installed using homebrew. Where do I find the setup.py or configure command? Thanks for the help!

    @dtrodrigues
    Copy link
    Mannequin

    dtrodrigues mannequin commented Feb 13, 2021

    You'll probably need to contact the pyenv project (https://github.com/pyenv/pyenv) if the following doesn't work for you. However, doing something like

    CFLAGS="-I$(brew --prefix)/include" CPPFLAGS="-I$(brew --prefix)/include" LDFLAGS="-L$(brew --prefix)/lib" pyenv install 3.9.1

    is working for me to use the lzma module if you already have xz installed with Homebrew.

    @richiejthomas
    Copy link
    Mannequin

    richiejthomas mannequin commented Feb 13, 2021

    That works for me. Thank you!

    On Fri, Feb 12, 2021 at 11:12 PM Dustin Rodrigues <report@bugs.python.org>
    wrote:

    Dustin Rodrigues <dust.rod@gmail.com> added the comment:

    You'll probably need to contact the pyenv project (
    https://github.com/pyenv/pyenv) if the following doesn't work for you.
    However, doing something like

    CFLAGS="-I$(brew --prefix)/include" CPPFLAGS="-I$(brew --prefix)/include"
    LDFLAGS="-L$(brew --prefix)/lib" pyenv install 3.9.1

    is working for me to use the lzma module if you already have xz installed
    with Homebrew.

    ----------


    Python tracker <report@bugs.python.org>
    <https://bugs.python.org/issue41116\>


    @ambv
    Copy link
    Contributor

    ambv commented May 2, 2021

    New changeset b29d0a5 by Ned Deily in branch '3.8':
    [3.8] bpo-41100: Support macOS 11 Big Sur and Apple Silicon Macs (bpo-25806)
    b29d0a5

    @ned-deily ned-deily removed the 3.7 (EOL) end of life label May 2, 2021
    @ned-deily ned-deily removed the 3.7 (EOL) end of life label May 2, 2021
    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    3.8 (EOL) end of life 3.9 only security fixes 3.10 only security fixes OS-mac
    Projects
    None yet
    Development

    No branches or pull requests

    3 participants