-
-
Notifications
You must be signed in to change notification settings - Fork 31.7k
build on macOS 11 (beta) does not find system-supplied third-party libraries #85288
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Comments
When building on macOS 11 (beta), a number of modules that should normally build on macOS fail because the system-supplied third-party libraries are not found. The necessary bits to build these optional modules were not found: The list should look like this (with no additional third-party libs supplied from another source like Homebrew or MacPorts): The necessary bits to build these optional modules were not found: The problem is due to a change in the 11 beta versus 10.15 or earlier systems: This breaks tests in setup.py using find_library_file() to determine if a library is present and in what directory it exists. setup.py depends on Lib/distutils/unixccompiler.py to do the dirty work. A similar problem arose on earlier macOS releases when header files could no longer be installed in the systems /usr/include; setup.py had to be taught to look in the SDK being used implicitly or explicitly by the compiler preprocessor. We could probably do something like that here while trying to avoid changes that might break downstream supplements/replacements to distutils, for example, setuptools. There is a workaround: explicitly specify the SDK location to ./configure (you also need to specify the universal archs setting): ./configure \ |
These build for me (master branch), because the support for .tbd files in the macOS SDK still works. The ones that don't build for me:
This was with a "Universal 2" build (which explains _ctypes) and without 3party libraries (which explains _gdbm, _ssl and _lzma). ossaudiodev and spwd are not supported on macOS even on older versions of the OS. I can get _decimal to build with a trivial patch to Modules/_decimal/mpdecimal.py (I'll create a PR later today). The PR will obviously be untested w.r.t. actually supporting ARM64. |
I should have specified that the failures I saw were just with the default intel-64 arch on the beta. After thinking about it a bit and reviewing the code, I think the best approach is to handle this in _osx_support.py as part of the compiler flags customization. What's been happening over the years is that Apple has been deprecating and then removing system header files and system library files on running systems and instead forcing the use of an SDK to build with for both. That's fine but, to continue to support a range of systems, we should be able to handle the various default configurations when the user does not explicitly specify a particular SDK to use. The Apple compiler chain does the right thing under the covers so that C compilations just work. setip.py's macosx_sdk_root() now has initialization code to discover the path to the system header files that the compiler is using; it calls the compiler with -v on a dummy program and then scans the compiler output for the include paths searched until it finds either /usr/include or a path ending in .sdk/usr/include. (It's a bit of a hack but a fairly widely-used hack that works with both clang and gcc, at least, and for the Apple-supplied compilers, takes into account the dynamic specification of SDK via xcrun.) I think the thing to do is move that sdk path detection into _osx_support and then, at the end of other customization steps, if there isn't already a valid -isysroot provided by the user (either through ./configure or CFLAGS et al, add a -isysroot for the discovered default SDK path (or /) into the appropriate customized compiler and linker flags. It would also provide a function to return the SDK path for setup.py (the equivalent of the current macosx_sdk_root()) and others to use. setup.py and Distutils should then do the right things for both header and library file paths on any configuration (whether header files are installed or not and whether lib files are installed or not, Command Line Tools or Xcode) and, with luck, any other Distutils replacements that also use _osx_support will also do the right thing. I had contemplated removing _osx_support at some point in the future but, now with the return of universal files, I'm glad we didn't. |
I see, and (as expected) get the same output when I build like that. I agree that we should teach _osx_support about this. I guess we should add a function that returns the SDK root, which can then be used in distutils (and setuptools, ...) instead of looking for -isysroot in CFLAGS. The new function can than look for the default SDK when there is no sysroot flag. For 3.10 we should look into cleaning up configure.ac, it contains some logic that is no longer valid or necessary (such as looking for the architectures supported by /usr/lib/libSystem.dylib). We should also avoid adding "-isysroot" to the CFLAGS/LDFLAGS to avoid problems when building on user machines that might have a different version of Xcode. |
So close... the problem we're running into is that "-isysroot" is only added for universal builds, not for regular builds. Furthermore unixccompiler doesn't know that is should always look in the SDK and not in the regular location (for system locations). I guess we should switch to:
The first item might affect older systems where Xcode shipped with 2 SDKs (current and previous OS version). Is that something we can back port to 3.8 and 3.9? A less clean, but smaller, change is to teach the configure script to always add -isysroot on macOS, not just when doing a universal build. |
With './configure; make' of master on macOS 11 on Apple Silicon I get the library detection failure. The quick and easy fix was to amend inc_dirs and lib_dirs in setup.py: if MACOS:
+ sysroot = macosx_sdk_root()
+ self.inc_dirs.append(sysroot + '/usr/include')
+ self.lib_dirs.append(sysroot + '/usr/lib') Interestingly the lzma library is found, but the module fails because it cannot find lzma.h, which does indeed appear to be missing in the SDK. |
I am running macOS Big Sur Version 11.1 on Silicon and still get the error about the missing lzma.h file. I can confirm that I use the latest Python 3.9.1 version which includes the patches of this issue. gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Wstrict-prototypes -Werror=implicit-function-declaration -fvisibility=hidden -I./Include/internal -I./Include -I. -I/usr/local/include -I/Users/dev/Downloads/Python-3.9.1/Include -I/Users/dev/Downloads/Python-3.9.1 -c /Users/dev/Downloads/Python-3.9.1/Modules/_lzmamodule.c -o build/temp.macosx-11.1-arm64-3.9/Users/dev/Downloads/Python-3.9.1/Modules/_lzmamodule.o
/Users/dev/Downloads/Python-3.9.1/Modules/_lzmamodule.c:16:10: fatal error: 'lzma.h' file not found
#include <lzma.h> Is this supposed to be still the case? Thank you! |
Unfortunately, Apple does not include the XZ library with macOS so, to build that module, you will need to supply a copy from another source, either build it yourself or from a third-party library provider like Homebrew or MacPorts. The Python Developer's Guid has some suggestions: |
Thanks for the help! I tried the instructions, without success. I installed xz through homebrew (which needs to be installed on Silicon under /opt/homebrew). I can confirm the existance of: /opt/homebrew/Cellar/xz/5.2.5/include/lzma.h I used CPPFLAGS and also modified the system_include_dirs variable in the setup.py file, in both cases lzma.h seems to be ignored.
Am I missing something obvious here? Thanks! |
If your 'brew --prefix' is /opt/homebrew, then setting CFLAGS="-I/opt/homebrew/include" CPPFLAGS="-I/opt/homebrew/include" LDFLAGS="-L/opt/homebrew/lib" when running ./configure appears to be sufficient to build the interpreter with lzma support once you've already done 'brew install xz'. |
I tried that before, also with your command, without success. I saw that the CFLAGS includes weren't forwarded to the compiler - I don't know if that is only an issue here on my system. So instead I modified the setup.py and added the directory manually. It now works. Thanks for your help!! |
Hello, can you give me a few more details around the workaround for this. I am using pyenv to install 3.9.1 and not sure how I would include the xz installed using homebrew. Where do I find the setup.py or configure command? Thanks for the help! |
You'll probably need to contact the pyenv project (https://github.com/pyenv/pyenv) if the following doesn't work for you. However, doing something like CFLAGS="-I$(brew --prefix)/include" CPPFLAGS="-I$(brew --prefix)/include" LDFLAGS="-L$(brew --prefix)/lib" pyenv install 3.9.1 is working for me to use the lzma module if you already have xz installed with Homebrew. |
That works for me. Thank you! On Fri, Feb 12, 2021 at 11:12 PM Dustin Rodrigues <report@bugs.python.org>
|
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: