Python needs to check existence of functions at runtime for targeting older macOS platforms

[Zorg]

BPO	34597
Nosy	@ronaldoussoren, @ned-deily
Superseder	bpo-41100: Support macOS 11 and Apple Silicon Macs

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2020-11-08.09:27:05.533>
created_at = <Date 2018-09-06.14:59:08.304>
labels = ['OS-mac', '3.8', 'type-bug', '3.7']
title = 'Python needs to check existence of functions at runtime for targeting older macOS platforms'
updated_at = <Date 2020-11-08.09:27:05.532>
user = 'https://bugs.python.org/Zorg'

bugs.python.org fields:

activity = <Date 2020-11-08.09:27:05.532>
actor = 'ronaldoussoren'
assignee = 'none'
closed = True
closed_date = <Date 2020-11-08.09:27:05.533>
closer = 'ronaldoussoren'
components = ['macOS']
creation = <Date 2018-09-06.14:59:08.304>
creator = 'Zorg'
dependencies = []
files = []
hgrepos = []
issue_num = 34597
keywords = []
message_count = 15.0
messages = ['324698', '324706', '324746', '324826', '324884', '325251', '325253', '325258', '325342', '325347', '325348', '325650', '325652', '379572', '380540']
nosy_count = 3.0
nosy_names = ['ronaldoussoren', 'ned.deily', 'Zorg']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = '41100'
type = 'behavior'
url = 'https://bugs.python.org/issue34597'
versions = ['Python 3.6', 'Python 3.7', 'Python 3.8']

[Zorg]

If one wants to compile Python and embed it in their applications, or more generally, deploy the framework / libraries to older systems, then Python needs to check the existence of functions at runtime that are unavailable to Python's supported minimum OS.

For example, Python makes use of:
clock_gettime
getentropy

These functions are only available on macOS 10.12. Python currently states it supports 10.9. Checking for their existence at compile time isn't good enough; compiling Python on 10.13 and deploying back to 10.9 or 10.11 for example will not work, and cause a crash at runtime. One can make sure the functions are weakly linked and check they aren't NULL to get around this.

There may be more such functions Python is using; I haven't extensively searched for all usages.

Alternatively, Python could state that it does not properly support embedding (probably not desirable), or Python could bump the minimum supported OS requirements in its documentation.

Else one might be able to compile Python by turning off such features? But the user needs to know what those are, Python gives no warnings when MACOSX_DEPLOYMENT_TARGET/--mmacosx-version-min are low enough, and Python wouldn't be able to use those functions when they are actually available on newer systems.

[ronaldoussoren]

The primary Python.org installer supports 10.9 and later by way of being compiled on 10.9, and there is also an installer that supports 10.6.

There currently is no promise that building on 10.13 results in code that works on earlier versions of macOS.

That said, I agree that building on later macOS releases and deploying to older ones is useful, and this is something that I used to do in the past (I currently have the good fortune to only need to deploy to macOS 10.13).

It is possible to add runtime checks for APIs, that's something I've done in the past to for the same reason, and that code is still present (for example in posixmodule.c).

Is this something you can and want to work on?

[Zorg]

The minimum OS target (which looks like 10.13) should ideally be well defined and tested using the latest SDK, rather than there being "no promises". Or at least it should be documented that this isn't currently supported, or how to disable specific features and which ones to disable. I downloaded the source tarball from https://www.python.org/downloads/release/python-370/

Python then isn't viable currently for embedding inside applications (https://docs.python.org/3/extending/embedding.html). 3rd party app developers on macOS typically have a policy at least supporting back to N-1 or N-2'th OS, and in many cases it shouldn't be their dependencies that drive this.

The other two functions require 10.13:
futimens
utimensat

Fortunately the -Wunguarded-availability warning is enabled which spits out warnings on which functions aren't available when targeting a min OS (MACOSX_DEPLOYMENT_TARGET)

Can't say I can contribute anything here unfortunately :(. I had to do something though, so I hacked the configure scripts and pyconfig.h to make sure the clock_* and the above functions I mentioned above weren't being used. getentropy was easy to fix more properly because it already has a fallback at runtime in case it's not working, so it was just an additional NULL check. Looks like the functions are weakly referenced when building to an older OS target.

A bad step in future direction would be to continue making "no promises" and start using functionality only available on 10.14, 10.15, etc without fallback.

[lazka]

related: bpo-31359

[ronaldoussoren]

Binaries currently work on the system you compiled on and any later release of macOS. That's the only "promise" there currently is.

Anything better requires work, and hence someone willing to do that work.

I'd love to be able to build on the latest macOS and deploy to older versions, but that's functionality I don't need myself at the moment which means this does not have priority for me.

BTW. I haven't checked yet how invasive supporting weak linking for these symbols will be. That depends on where the symbols are used.

[ned-deily]

While, as Ronald notes, it would be nice to fully support weak linking so that building Python on macOS 10.n with a deployment target for 10.m where m < n so that an application with an embedded or bundled Python can run on 10.m through 10.n systems, in the mean time there is a simple and safe alternative that has been used for years: build the Python on 10.m (or use a ready-built Python built on 10.m or lower). That's what we do for the macOS Pythons provided by python.org installers, e.g. the installer variants that claim 10.9+ support are actually built on 10.9. If you need an app bundle with an embedded Python, py2app can greatly simplify the process of constructing one.

[Zorg]

https://bugs.python.org/issue31359 has a better details about this issue.

I personally believe it's safer to link against a newer SDK and deal with the trouble of finding out what to strip out, which will be opted into implications or code paths that linking a binary against a newer SDK may provide.

[ned-deily]

https://bugs.python.org/issue31359 has a better details about this issue.

Yea, the approach there is again dealing with using a newer SDK to build for an older system.

I personally believe it's safer to link against a newer SDK and deal with the trouble of finding out what to strip out, which will be opted into implications or code paths that linking a binary against a newer SDK may provide.

Why do you believe that is safer? Apple goes to great lengths to provide compatibility for existing applications to keep running on newer systems. That's not to say that eventually certain APIs are deprecated and removed but Python itself makes fairly minimal demands on macOS, i.e. it pretty much does not use anything other than basic libc-style system calls. The one obvious disadvantage is that Python apps can't take advantage of newly introduced system calls (introduced after 10.m) when running on those newer systems. I honestly don't recall any case dating back to 10.5 or so where there have been any reported problems with running a 10.m Python on a 10.n system (as best as I can recall!). On the other hand, trying to support weak linking and backward compatibility requires more code and significant increase in ongoing testing. It would be a good thing to have but there is a significant cost and added risk to do so.

[ronaldoussoren]

Linking with newer SDKs is needed to get access to some newer features and APIs. That's less relevant for Python itself, but can be relevant for the embedding application. You can still compile Python itself on an older machine, then link it into an application build against a newer SDK. This does make the build process harder though.

However, there seem to be some issues when you mix code linked for different deployment targets, for example <https://bitbucket.org/ronaldoussoren/pyobjc/issues/174/crash-using-dictionaryservices-on-1012\>. This is issue contains a C extension that works correctly when CPython is build with a new SDK and crashed hard with the interpreters on Python.org.

This is one reason why I will at some time in the future look into this, but I cannot say when that will happen because I have limited time to work on this.

[Zorg]

Why do you believe that is safer? Apple goes to great lengths to provide compatibility for existing applications to keep running on newer systems.

They also don't want developers to develop using older SDKs. Hypothetically speaking, more information may be stored in the binaries, and there could be something down the road to opt into that requires all binaries to be linked against a certain SDK. Or your code takes an older path based on the SDK version at runtime -- python may include some other higher level baggage too, like the tkinter stuff for instance.

There is some middle ground other than weak linking; stripping out features from the configure part of the project and opt in to using the deprecated APIs. Not as ideal as weak linking, but better than building on an older SDK IMO.

[Zorg]

Also more obviously: it's less convenient for others keeping themselves up to date to set up an older development environment.