Support builds on macOS without installed system header files

[dimpase]

BPO	36231
Nosy	@ronaldoussoren, @ned-deily, @embray, @jdemeyer, @dimpase
PRs	bpo-36231: no "proper" header files on macOS 10.14 Mojave #12825 bpo-36231: Support building on macOS without /usr/include #13773 [3.7] bpo-36231: Support building on macOS without /usr/include (GH-13773) #14208 [2.7] bpo-36231: Support building on macOS without /usr/include (GH-13773) #14256
Files	noincludedirs_OSX.patch: POC patch for "no-headers" OSX

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/ned-deily'
closed_at = <Date 2019-06-20.06:46:28.503>
created_at = <Date 2019-03-08.10:03:47.265>
labels = ['OS-mac', '3.8', 'build', '3.7']
title = 'Support builds on macOS without installed system header files'
updated_at = <Date 2019-06-20.08:22:24.734>
user = 'https://github.com/dimpase'

bugs.python.org fields:

activity = <Date 2019-06-20.08:22:24.734>
actor = 'dimpase'
assignee = 'ned.deily'
closed = True
closed_date = <Date 2019-06-20.06:46:28.503>
closer = 'ned.deily'
components = ['Build', 'macOS']
creation = <Date 2019-03-08.10:03:47.265>
creator = 'dimpase'
dependencies = []
files = ['48192']
hgrepos = []
issue_num = 36231
keywords = ['patch']
message_count = 13.0
messages = ['337461', '337540', '337680', '337906', '337912', '340202', '344407', '346008', '346049', '346062', '346101', '346104', '346108']
nosy_count = 5.0
nosy_names = ['ronaldoussoren', 'ned.deily', 'erik.bray', 'jdemeyer', 'dimpase']
pr_nums = ['12825', '13773', '14208', '14256']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'compile error'
url = 'https://bugs.python.org/issue36231'
versions = ['Python 2.7', 'Python 3.7', 'Python 3.8']

[dimpase]

Neither Xcode nor its command-line tools on macOS 10.14 Mojave come with header files installed in /usr/ and other "normal" directories.

This is not documented in https://devguide.python.org/setup/#macos-and-os-x

While an extra step to handle this, i.e. to install the headers, is available (see a discussion on https://bugs.python.org/issue34956), Apple stated that this workaround will disappear.

It is thus highly desirable to provide a way to deal with headers located not at /usr/include, but at xcrun --show-sdk-path/usr/include.
A small change in setup.py along the lines of the following:

--- a/setup.py
+++ b/setup.py
@@ -134,7 +134,8 @@ def macosx_sdk_root():
     cflags = sysconfig.get_config_var('CFLAGS')
     m = re.search(r'-isysroot\s+(\S+)', cflags)
     if m is None:
-        sysroot = '/'
+        import subprocess
+        sysroot = subprocess.check_output(["xcrun", "--show-sdk-path"]).decode("utf-8").strip('\n')
     else:
         sysroot = m.group(1)
     return sysroot
@@ -146,6 +147,7 @@ def is_macosx_sdk_path(path):
     """
     return ( (path.startswith('/usr/') and not path.startswith('/usr/local'))
                 or path.startswith('/System/')
+                or path.startswith('/Applications/')
                 or path.startswith('/Library/') )

with the necessary changes to enable various modules (see attachment for a complete POC diff against the current master), the result builds and passes all the (enabled) tests on an OSX 10.13 system with empty /usr/include and /usr/local/include containing only LZMA headers.

Needless to say, a proper patch would not modify Modules/Setup, it'd adjust configure.ac etc.

[dimpase]

Needless to say, subprocess is most certainly an overkill, something less involved would do the job, without the need for all the module dependencies of subprocess.

[embray]

Perhaps it would be better if the xcrun --show-sdk-path thing were run at configure-time and its result shoved into a variable we can read with sysconfig.get_config_var()

[dimpase]

I won't mind to provide a PR for this. but it is not clear what the goal should be. Is it to build a working OSX Python with as few external to Xcode deps (it seems that only lzma is needed) as possible?

[ned-deily]

Thanks for the report and for the PR offer but let's hold off on that for the moment: I'm planning to merge a somewhat different approach.

[dimpase]

In case,I have opened PR #12825
to provide our solution to this issue.

[ned-deily]

New changeset 0288dd6 by Ned Deily in branch 'master':
bpo-36231: Support building on macOS without /usr/include (GH-13773)
0288dd6

[ned-deily]

New changeset c730211 by Ned Deily in branch '3.7':
bpo-36231: Support building on macOS without /usr/include (GH-13773) (GH-14208)
c730211

[dimpase]

I find it puzzling that a relatively clean and short solution using autotools and the Apple-approved way to get the location of the headers is rejected, and a hacky, longer way that scrapes output of the compiler is being pushed instead.

[ned-deily]

Thank you again for the suggested PR. Using "xcrun --show-sdk-path" at configure time *is* appealing. Unfortunately, it does not cover all of the necessary use cases.

One, the --show-sdk-path option is not available on old versions of xcrun, versions we still build with to support older releases of macOS. Second, some users build Python for macOS with compiler tool chains other than the Apple-supplied ones (for example, current gcc) that do not necessarily support the transparent selection of header and library files location via xcrun and friends. Third, capturing the selected SDK path at configure time is no guarantee that the same SDK path will be used for extension module builds when setup.py runs. With the Apple-supplied tools, the actual SDK path used is determined each time the compiler front-end is invoked and depends on the then-current selected values (e.g. the most recent value set by 'xcode-select --switch') and the current value of environment variables (e.g. like DEVELOPER_DIR and SDKROOT). Or a different or non-Apple compiler could now be in use by overriding CC. In other words, lots of edge cases largely due to the inherent flexibility of Apple's compiler frontend.

The other issue here is the behavior of setup.py in trying to make reasonable default choices for finding header and library files for the extension modules it is building; to do so, it tries to guess what the compiler frontend is going to do and that's the real hack. Life would be much simpler if Python relied on a modern autotools build setup - although some of the same issues of dynamic SDK locations would still apply but at least they would apply consistently - and not the legacy mixture of build tools we have today. Unfortunately, it would be a big deal to replace the current build system and, while it would be desirable, that's a very big project.

One other point: while scraping the output of the compiler is hacky, it is a well-known and widely-used technique and is already in use elsewhere in setup.py; this code was adapted from that in add_multiarch_paths().

[ned-deily]

New changeset c421c66 by Ned Deily in branch '2.7':
bpo-36231: Support building on macOS without /usr/include (GH-13773) (GH-14256)
c421c66

[ned-deily]

Thanks again everyone. The general solution has now been merged to all active branches for release in 3.8.0, 3.7.4, and 2.7.17.

Note that bpo-19960 identifies an additional problem only on 2.7 where a few extension modules, notably zlib, are still not being built when header files are not installed; a PR for that issue is currently awaiting review.

[dimpase]

Thanks for the review of our PR. I think not all points made there are relevant:

• on older systems, where xcrun --show-sdk-path is not available, one can assume that /usr/include is there, and simply set the corresponding variable to this value in ./configure

• building Python with a non-system compiler is only possible with system headers, and thus either SDK or at least command line tools, available, and all is well.

• mid-flight switching to a different SDK/compiler is dodgy, and in particular switching to a different set of headers is asking for trouble. And the gcc toolchain is not much different in this way - e.g. on Gentoo one can do gcc-config to switch from one profile to another, with provably "interesting" results if it is done mid-flight.

• the Python 2.7 zlib issue is not a problem with our PR.

---------

Most probably, the desired switch to autotools can only happen gradually, by moving more of setup.py to ./configure, and our PR is a step in such direction.