gh-114099: Add configure and Makefile targets to support iOS compilation.

[freakboy3742]

Part of the PEP 730 work to add iOS support.

This PR adds the iOS targets to configure and Makefile to support compiling an iOS framework. It does not include the test harness and related targets that was included in #115063 - that will come in a follow up PR.

This PR slightly contains one deviation from PEP 730, as it uses the <cpu>-<abi> format for PLATFORM_TRIPLE and _PYTHON_HOST_PLATFORM, rather than <abi>-<cpu> (i.e., it uses arm64-iphoneos, not iphoneos-arm64). This was identified during the review of #115120 as an inconsistency with the ordering of existing platform triples. This is an almost entirely cosmetic change with no compatibility considerations, as the only impact is the final naming scheme of sysconfig modules and the sys.implementation._multiarch attribute.

As of this PR, it is now possible to compile an iOS framework; but the framework still isn't usable at runtime.

Summary of changes

• Patches config.sub to include 2 modifications required to support iOS/tvOS/watchOS builds (supporting the arm64_32 CPU identifier, and the -simulator suffix). These were submitted upstream as part of a patch in August last year, but that patch was only partially applied. The autoconf contribution process is opaque, and no feedback was given on why the other parts were rejected (there wasn't even feedback that they'd been partially accepted - I discovered that by accident). The extra pieces have been re-submitted upstream.
• Modifies the internal target name for setting up a versioned macOS framework, and adds a set of targets for an unversioned iOS-style framework. See below for details on iOS framework structure.
• Adds the flags to build iOS-compatible binary modules linked against the Python framework
• Presumptively adds the arm64 device and simulator targets to the Tier 3 support list. Getting an x86_64 buildbot is proving difficult because of hardware availability, so I've omitted that target from Tier 3. This possibility was flagged in PEP 730.
• Skips the automated detection of a couple of system functions where autoconf gets the answer wrong - autoconf only checks whether a symbol can be linked, and doesn't use header files; iOS headers raise warnings when some symbols are used. There are also methods that exist and can be compiled, but raise errors if used at runtime.
• Automatically skips the configuration of ac_cv_file__dev_ptmx and ac_cv_file__dev_ptc, which are usually mandatory command line arguments for cross-platform builds. iOS is a reliable cross-build target, so we know these features aren't available.
• Adds an iOS resources folder that contains template files and stub binaries. See below for a discussion about the stub binaries. Internal documentation for building the iOS framework has been added; user-facing docs will be added in a future PR.
• Includes a pyconfig.h file that can be used to switch between CPU-dependent pyconfig.h files.
• Disables the standard library modules that aren't available on iOS.
• Doesn't include an tvOS and watchOS targets or resources. These will be submitted independently.

Differences between macOS and iOS Frameworks

iOS frameworks are slightly different to macOS frameworks.

• macOS frameworks are "multi-version", as they are designed to support having multiple framework versions in a single Framework folder. iOS frameworks can't be shared between apps, so they don't support "multi-version" format.
• The metadata required by Info.plist is slightly different, and must be in the root of the framework, not a Resources subfolder.
• The install name for the library in the framework must be based on an @rpath, rather than an absolute path, as the install path for the framework will be app-specific.
• iOS frameworks can only include binary .dylib artefacts and headers; and the headers won't be copied into the final distribution binary. As a result, the standard library and support binaries cannot be distributed inside the framework - they must be distributed parallel to the Framework folder.

To accomodate these difference, the make install target for iOS will generate a file structure like the following:

(--enable-framework location)

• bin
• include symlink to -> Python.framework/Headers
• lib
  • python3.X
    • ...
• Python.framework
  • Headers
    • ...
  • Python (the dylib binary)
  • Info.plist

The task of copying the standard library into an appropriate location in the final app bundle is left as a task for the developer using the framework.

Stub binaries

Xcode doesn't expose explicit compilers for iOS/tvOS/watchOS; instead, it uses an xcrun script that resolves to a full compiler path (e.g., xcrun --sdk iphoneos clang to get the clang for an iPhone device). However, using this script poses 2 problems:

• The output of xcrun includes paths that are then machine specific, resulting in a sysconfig module that cannot be shared between users
• It results in CC/CPP/LD/AR definitions that include spaces. There is a lot of C ecosystem tooling (including distutils and setuptools) that assumes that you can split a command line at the first space to get the path to the compiler executable; this isn't the case when using xcrun.

To avoid these problems, the iOS/Resources/bin folder includes shell script wrappers around the tools needed by configure, named using the scheme that autoconf expects by default. These scripts are relocatable, and will always resolve to the appropriate local system paths. By including these scripts in the "installed" bin folder, the contents of the sysconfig module becomes useful for end-users to compile their own modules.

• Issue: Add support for iOS as a target platform #114099

[mhsmith]

I don't have access to resolve conversations in this repository, so I've marked them with a thumbs up instead. The only outstanding one, which we need guidance from the core team on, is whether the iOS files should go in a top-level directory as this PR currently does, or a subdirectory of Tools.

[erlend-aasland]

[...] The only outstanding one, which we need guidance from the core team on, is whether the iOS files should go in a top-level directory as this PR currently does, or a subdirectory of Tools.

I personally would prefer one top-level directory, like the current Mac/ dir. Windows stuff is split between PC/ and PCbuild, which IMO is unfortunate. I haven't looked at how the WASM stuff is organised.

[freakboy3742]

[...] The only outstanding one, which we need guidance from the core team on, is whether the iOS files should go in a top-level directory as this PR currently does, or a subdirectory of Tools.

I personally would prefer one top-level directory, like the current Mac/ dir. Windows stuff is split between PC/ and PCbuild, which IMO is unfortunate. I haven't looked at how the WASM stuff is organised.

That's good enough for me. I've modified the docs to use a folder inside iOS.

[erlend-aasland]

Thanks to @mhsmith for the thorough reviews. Waiting thumbs-up from @ned-deily before landing.

[ned-deily]

Overall, this is a very nice PR and a pleasure to review. I don't have time left tonight to actually test it; I will do so tomorrow. But I do have a few, relatively minor review comments here. Let me know what you think.

[ned-deily]

The use of these stub scripts for executables is a clever solution. As they stand, I think there is one potential shortcoming if I understand the xcrun documentation correctly (I haven't yet verified this): according to the man page, specifying --sdk to xcrun overrides any SDKROOT environment variable, something which I've had occasion to use when building and testing for multiple macOS versions. It's probably less of a deal on iOS but I think it would be nice to allow for this. If the behavior is as documented, perhaps the stub scripts could do a test for the presence of an SDKROOT env variable and, if defined, not specify --sdk iphoneos?

[freakboy3742]

That's an easy enough change to make; however, it does open a potential class of bugs.

As currently defined, arm64-apple-ios-clang will always be a clang install for ARM64 iOS devices. If we allow for SDKROOT as you describe, a user could easily think they're compiling an iOS build, but actually be using an iOS simulator SDK (or worse - a macOS or watchOS SDK).

This is especially problematic when the values are embedded in sysconfig, as the CC value embedded in sysconfig will become dependent on the end-user's environment.

You can still switch between different Xcode installs with xcode-select for testing purposes; I'd argue this is likely more useful for testing purposes, as Xcode leans heavily to having a single iOS SDK per Xcode install.

My inclination is that being explicit with --sdk in these scripts and overriding SDKROOT is preferable on balance, but I'll defer to your judgement if you feel strongly otherwise. If we do make this change, I'll add some docs clarifying how SDKROOT is interpreted, and clarifying that it's the user's responsibility to make sure SDKROOT matches their intended compilation target.

[ned-deily]

I see two drawbacks with relying solely on xcode-select. One, xcode-select -s has effect system-wide, and so would affect all other uses and users of Xcode on that system. Of course, there is another way to effect this on a process basis: set a DEVELOPER_DIR env variable. But, two, even with that, AFAICT there still would be no way to select a non-default iOS SDK within a particular Xcode instance and, while it may not be the default case at the moment, it would be better to not preclude supporting more than one SDK available in an Xcode instance. I think the primary value of supporting all this would be during development and testing of Python iOS frameworks themselves and, as such, you likely wouldn't want to use them in an actual release of a Python iOS framework but there might be cases where you would. (And, the end-user of the framework, presumably someone developing an iOS app, would need to ensure they had a compatible developer environment anyway.)

So perhaps a slightly simpler way to address these concerns would be to define a new env variable that would allow overriding the SDK name in the stubs, something like IOSSDK='iphoneos17.1' (I'm not hung up on the name) instead of the default iphoneos ?, and mentioning setting DEVELOPER_DIR to use a non-default Xcode installation:

xcrun --sdk ${IOSSDK:-iphoneos} clang -target arm64-apple-ios $@

Or we could just defer this until a demonstrated need arises but I think it is worth considering.

[freakboy3742]

Passing in a specific SDK (or even better, a specific SDK version) with a custom environment variable makes sense to me. I'll push an update shortly that uses IOS_SDK_VERSION so you can get iphoneosX.Y and iphonesimulatorX.Y from a single environment variable (since the base names of the SDKs are stable, and I can't think of a use case where you'd want a different version for the device vs simulator SDK in a single build), along with docs covering the new variable and DEVELOPER_DIR

[ned-deily]

Sorry, I didn't notice these earlier: all of the stubs should just use a shebang line of /bin/sh rather than bash since bash is officially deprecated on macOS, and there are no bash-isms used.

[freakboy3742]

@ned-deily As requested, an update that adds:

• Support for an IOS_SDK_VERSION environment variable in the compiler wrapper scripts
• Documentation of the IOS_SDK_VERSION option, plus the DEVELOPER_DIR and xcode-select options for selecting a different Xcode install
• Removal of bash usage
• Introduction of DESTDIR for iOS framework installs.

[ned-deily]

When building outside of the source directory, the copy operation of the stub scripts to the bin directory fails because they are only in the source directory.

for file in $(srcdir)/$(RESSRCDIR)/bin/* ; do \

[ned-deily]

Thanks for the requested changes. The DESTDIR support looks good. While testing building with a separate build directory (a commonly-used configuration), there was a minor problem with copying of the stub scripts, noted above. I haven't finished testing and tracking down a few other issues yet but there does seem to be a missing dependency in the Makefile as the separate build directory build fails for me with make -j4 when reaching the extension module builds:

arm64-apple-ios-simulator-clang -dynamiclib -F . -framework Python -mios-version-min=12.0     Modules/arraymodule.o   -o Modules/array.cpython-313-iphonesimulator.dylib
ld: framework 'Python' not found
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make: *** [Modules/array.cpython-313-iphonesimulator.dylib] Error 1

It doesn't fail with just a single-process (no -j) make.
More later.

[freakboy3742]

... there does seem to be a missing dependency in the Makefile as the separate build directory build fails for me with make -j4 when reaching the extension module builds:

This sounds familiar... I seem to recall @mhsmith mentioning a similar issue with Android builds. Not sure if it's the same cause.

FWIW: I've historically avoided -j because Python's build system has broken several times in the past for cross-platform builds. It's easy for bugs of that nature to slip in, especially when I don't think any cross-platform builds are part of the CI matrix. Maybe this iOS and Android work can indirectly help to address that :-)

I'll investigate and work out what is going on.

[freakboy3742]

@ned-deily I've think found the issue with parallel builds - turns out the issue was iOS specific in this case.

I've also addressed the srcdir issue.

[mhsmith]

For Android the parallel build issue was fixed in #115780, which I see you've already found.

[ned-deily]

Almost there! See the two comments in the review. Otherwise, everything looked good! I didn't do an exhaustive build test but I think I covered the most interesting cases.

FWIW, I'm attaching a script I used to test builds. It does not attempt to cover everything nor build working libraries (for example, no third-party libs) but it does go through the motions of building both a macOS build Python and then building all three supported iOS archs plus a rudimentary test of using DESTDIR. I ran it successfully on macOS 14.3.1 with Xcode 15.2 on both Apple Silicon and Intel Macs. I also successfully tested building using an Xcode pre-release and explicit SDK:

export DEVELOPER_DIR=/Applications/Xcode-beta.app
export IOS_SDK_VERSION=17.4
cd /path/to/srcdir
./build-python-ios-frameworks.sh

I'm attaching the script here just for future reference rather than suggesting including it in the PR. Perhaps it might be useful as a start for a buildbot or CI step.
build-python-ios-frameworks.sh

[ned-deily]

Because the Info.plist is marked as a config file in configure.ac (AC_CONFIG_FILES([iOS/Resources/Info.plist]), autoconf ensures that configure copies that file into the build directory, so the change to add $(srcdir) here is not needed here (and causes an error) unlike the $(srcdir) below for the stub scripts which are not config files which does now work correctly (thanks!).

[ned-deily]

One further thought here: perhaps we should be a bit more exact about what is meant by the *exact* same version. A builder could rightfully question whether that means that the build Python has to be built from the exact same source checkout as the iOS version(s) being cross-built. Or is a build Python from the same branch OK, e.g. a Python 3.13.x? This is more of an issue once 3.13, say, is released and we promise things like ABI compatibility, etc.

[freakboy3742]

Once a release is finalised, it's generally safe to use any 3.X build as a host Python for a cross platform build of 3.X.Y. However, there were at least 2 commits in January (both threading related) that caused an issue with iOS cross-platform builds of 3.13 if the host Python wasn't the same commit hash. That's easy enough to put down to "main branch is a moving target"; but I believe this could manifest with 3.X.Y as a host for 3.X.Y+1 because there's no guarantee that the bytecode magic number will remain consistent in a 3.X series.

So - the safest answer here is "the same commit hash"; but there's often a bit of leeway. I'll clarify the language here to a "hope for the best, expect the worst" kind of framing.

[freakboy3742]

@ned-deily Updated with a README clarification and $(srcdir) cleanup.

[ned-deily]

Looks good to me. Thanks again for the attention to detail. If any other comments come up, they can be resolved in a follow-on PR. Time to move on to the next phase.

[mhsmith]

I'm attaching the script here just for future reference rather than suggesting including it in the PR. Perhaps it might be useful as a start for a buildbot or CI step.

I've already put together a more production-ready script for Android, based on the existing WASI build script from Tools/wasm/wasi.py, so I suggest iOS follows the same pattern.

The current version is here, and I'll submit it in a PR as soon as the lower-level Android build fixes have been merged.