gh-140189: Add CI job to test iOS builds.

[freakboy3742]

Adds a CI task to build and test an iOS XCframework.

The shape of this is almost identical to the Android task; the only notable difference is that it explicitly selects an Xcode version. Explicitly selecting an Xcode version is recommended by GitHub for all macOS activity, but changes rolled out to GitHub Actions during August made this a necessity, as the default Xcode configurations will not reliably start an iOS simulator (see actions/runner-images/issues/12541 and actions/runner-images/issues/12751).

• Issue: iOS CI and binary releases #140189

[freakboy3742]

Hrm... this works, but I get the impression something isn't quite working right with logging, because the group collapsing ins't working the same as it is on Android.

@mhsmith Any ideas? It looks like the output of the subprocesses is being piped entirely differently...

[hugovk]

This job takes about 34 minutes, making it the new bottleneck. The other slowest are free-threaded Windows on x64 (26m) and arm64 (24m):

Is there anything we can do to speed it up a bit?

It's running tests sequentially in a single process, which takes 17 minutes. Can we parallelise? Or cache any of the build process?

[StanFromIreland]

Is it possible to build without installing, maybe that would save some time? From the logs it looks like everything is installed:

/usr/bin/install -c -m 644 ../../Lib/__future__.py Apple/iOS/Frameworks/arm64-iphoneos/lib/python3.15

Also looking at the logs, I am not familiar with the iOS build system, but it looks as if configure is ran four or five times, is that intentional?

[mhsmith]

Is it possible to build without installing, maybe that would save some time? From the logs it looks like everything is installed

The Android CI also runs make install, so that doesn't take much time in itself. But it's possible that the sheer quantity of output is slowing things down, and it certainly makes the logs more difficult to navigate. The make install output is very verbose and very unlikely to be useful, so the Android script suppresses it by default.

[freakboy3742]

Is it possible to build without installing, maybe that would save some time? From the logs it looks like everything is installed:

/usr/bin/install -c -m 644 ../../Lib/__future__.py Apple/iOS/Frameworks/arm64-iphoneos/lib/python3.15

Yes - running make install is part of the build process for iOS, because files need to be in the "installed" location in a framework in order to be used at all. It's not possible to use the "build in-situ" location, because the testbed app requires a formal framework.

However, as @mhsmith notes - this is a fairly minor component of the overall build time.

Also looking at the logs, I am not familiar with the iOS build system, but it looks as if configure is ran four or five times, is that intentional?

Four builds are very much intentional:

1. A build for macOS (for cross build purposes)
2. A build for arm64 iOS devices
3. A build for arm64 simulators
4. A build for x86_64 simulators

The first of those could be shared between iOS and Android (and be used for any other cross builds), and could also be used for actual macOS testing - although reusing in that way would require a fairly substantial rework of existing build and test workflows.

[freakboy3742]

This job takes about 34 minutes, making it the new bottleneck. The other slowest are free-threaded Windows on x64 (26m) and arm64 (24m):

Is there anything we can do to speed it up a bit?

It's running tests sequentially in a single process, which takes 17 minutes. Can we parallelise? Or cache any of the build process?

As with Android, the test suite isn't parallelizable on iOS, because we can't use subprocesses.

I'm not sure what we'd cache to speed things up - did you have something specific in mind?

The other obvious option would be "run fewer tests" - but that's not exactly satisfying.

I presume the concern is wall clock end-to-end time, rather than CPU time - if that's the case, the build could be parallelized in 2 phases - the three platform builds could be done in parallel; that would trim maybe 8 minutes from the overall end-to-end test time.

The other option would be for a CI build to only build and test single architecture (arm64-simulator). That would entirely remove the 8 minutes needed to build arm64-device and x86-64 simulator. This is essentially what the buildbots are doing today; they're able to complete CI end-to-end in about 22 minutes. However, that would mean that a second workflow would be needed for release purposes, and the release artefact wouldn't be tested until release day.

The other interesting detail that is that disk performance seems to be a factor here. On my local (4 year old) machine, I can complete 4 builds, sequentially, in 11 minutes. The same step takes 16 minutes in CI. There's a bunch of known issues with simulator performance on the macOS-15 image; those issues all seem to stem from disk I/O performance in the macOS hypervisor. I'm wondering if the same thing might at play here. I'll do some experiments to see if the macOS-14 or macOS-26 image are any better in terms of performance.

Not sure which of those options are more palatable.

[freakboy3742]

It's also worth noting that I can do a full end-to-end ci build, with the same options, on my 4 year old MacBook, in about 22 minutes 24 minutes. I'm not sure what is causing the slowdown in CI...

[mhsmith]

the group collapsing ins't working the same as it is on Android. @mhsmith Any ideas?

The script's own output is appearing after the output of its subprocesses, which probably means it's being buffered. The Android script works around this as follows:

cpython/Android/android.py

Lines 868 to 872 in e4f6445

	# Under the buildbot, stdout is not a TTY, but we must still flush after
	# every line to make sure our output appears in the correct order relative
	# to the output of our subprocesses.
	for stream in [sys.stdout, sys.stderr]:
	stream.reconfigure(line_buffering=True)

the three platform builds could be done in parallel

I guess that would mean a pipeline of 3 jobs:

• macOS build
• iOS build (matrix x3)
• iOS package and test.

Each job would pass an artifact to the next, but the number of files in each artifact would be significant, which could slow things down. Since macOS runners are no longer scarce in this project, it may be worth doing the macOS build in the same job as the iOS build, even if that means it's done the same way 3 times.

The timings will be be easier to analyze once the output grouping is fixed (press Shift-T in the log view).

[freakboy3742]

the group collapsing ins't working the same as it is on Android. @mhsmith Any ideas?

The script's own output is appearing after the output of its subprocesses, which probably means it's being buffered.

Yup - that did the job. Thanks.

I've also done some testing of the older macOS-14 image, and it's not significantly faster. The real speed fix was to not run --slow-ci.

the three platform builds could be done in parallel

I guess that would mean a pipeline of 3 jobs:

Yes, that was the sort of thing I had in mind.

However, I think I might have been overthinking the CI build requirements. CI isn't the only place tests are run; we also have buildbots, and the build process for release is an independent workflow. So - CI really only needs to be a "smoke test" to ensure that changes don't break iOS, not a full check over all architectures. To that end, we can configure the build so that:

• In CI, do a "thin" build - only compile the build Python, plus the Python the simulator on the current architecture, with --fast-ci
• On Buildbot, do a "fat" build - the full framework, over all required slices, with --slow-ci
• On release, also do a "fat" build with --slow-ci.

That means CI is able to complete in ~20 minutes with only 2 builds (essentially the same as Android); the full build on buildbots and release will take ~35 minutes. This is a little slower for buildbots (they're effectively doing --slow-ci on a "thin" build right now), but buildbot builds aren't as resource critical as CI; and I presume a slightly slower build is acceptable for releases.

There's some work needed to ensure that buildbots can use this new option (we need to be able to have different build instructions for 3.13 buildbots because the new build script has been backported), but that was already on my todo list to as part of rolling out the new build script.

@hugovk Is this more acceptable from a CI resource usage perspective?

[hugovk]

Yes, much better, thanks.

That means CI is able to complete in ~20 minutes with only 2 builds (essentially the same as Android); the full build on buildbots and release will take ~35 minutes.

Yep, release builds currently take about half an hour anyway, and bugfix releases with installers need to wait a few hours for those as well.

[freakboy3742]

Ok - I'm not sure why iOS has decided now is the time to start reliably failing a socket server test with a timeout... I'll have to investigate.