Compile in universal2 mode for macOS Python >= 3.9.1. #114

bwoodsend · 2021-09-23T17:18:47Z

For Python versions that support it, build in universal2 mode so that package authors can build macOS M1 compatible wheels. As it is at the moment, I've made the switch mandatory because I can't think of any reason (minus the increase in size) that a universal2 Python would cause any harm.

bwoodsend · 2021-10-01T14:04:17Z

@miketimofeev Prod prod...

miketimofeev · 2021-10-01T14:05:19Z

@bwoodsend sorry, we didn't have a chance to take a look yet, but it's on our radar

astrofrog · 2021-10-05T14:52:40Z

Just a +1 for this - it would be great to have!

miketimofeev · 2021-10-18T07:45:13Z

@bwoodsend we have announced macOS-10.14 image deprecation so in the scope of switching python packages generation to macOS-10.15 we will consider adding this change as well

bwoodsend · 2021-10-18T14:49:31Z

Hmm, not entirely sure what that implies but it sounds like good news ... 🙂

bwoodsend · 2021-11-25T08:27:39Z

azure-pipelines/build-python-packages.yml

@@ -6,7 +6,7 @@ stages:
 - stage: Build_Python_MacOS
  dependsOn: []
  variables:
-    VmImage: 'macOS-10.14'
+    VmImage: 'macOS-10.15'


One side effect of bumping the build image version is that, because the MACOSX_DEPLOYMENT_TARGET isn't set anywhere, the Pythons and therefore any wheels produced will only be compatible with macOS >= 10.15. Is there a reason MACOSX_DEPLOYMENT_TARGET isn't used? The python.org builds use minimum macOS version from 10.6 to 10.9 (which admittedly is overkill).

Due to `setup-python@v2` limitations, only x64 binary are created for macOS, universal binaries will be available once the following PR is merged: actions/python-versions#114

TonyXiang8787 · 2022-01-28T19:35:54Z

builders/macos-python-builder.psm1

+        ### For Python versions which support it, compile a universal2 (arm64 + x86_64 hybrid) build. The arm64 slice
+        ### will never be used itself by a Github Actions runner but using a universal2 Python is the only way to build
+        ### universal2 C extensions and wheels.
+        if ($this.Version -ge "3.9.1") {


May I ask why the Python version need to be greater than 3.9.1? In official Python download page, you can have universal Python for version 3.8.10. See below:

https://www.python.org/downloads/macos/

Yeah, you're right. I hadn't realised that they backported to Python 3.8.x as well. Note though that Python 3.9.0 does not support universal2 so the conditional would have to be:

if ($this.Version -ge "3.8.10" && $this.Version -ne "3.9.0") {

(assuming that that is the correct powershell syntax).

I think 3.8.10 is the only 3.8 version with universal2. So a ==3.8.10 or >=3.9.1 will work I guess.

It can compile universal2 for 3.8.11 - it's just by that point, python.org stops providing precompiled installers of any kind.

Using `universal2` Python environments is currently the only way to build Python wheels which support macOS M1.

bwoodsend · 2022-05-15T18:06:38Z

That last change was just a rebase plus adding Python 3.8.10+ to the Python versions to use universal2.

misl6 · 2022-05-31T21:07:17Z

As a bonus point:

Github is now providing actions/runner binaries for Apple Silicon (self-hosted)
A universal2 compatible Python, provided by actions/python-versions will also run on Apple Silicon.
With some (minimal?) changes (e.g: versions-manifest.json) a runner on Apple Silicon will likely be able to download and install the proper Python version, without any additional M1-specific configuration.

bwoodsend · 2022-06-01T08:16:18Z

Github is now providing actions/runner binaries for Apple Silicon (self-hosted)

That is the best piece of news I've heard for ages. I've been facing using Jenkins indefinitely at the day job since it's the closest we can get to CI/CD on M1.

bwoodsend · 2022-06-01T08:20:08Z

Github is now providing actions/runner binaries for Apple Silicon (self-hosted)

I take it that there's still no VMware in there so no job isolation and therefore no chance of this being available without self-hosting?

ssbarnea · 2022-06-06T11:18:08Z

Thanks! I read the code node and that comment worries me a little "The arm64 slice will never be used itself by a Github Actions runner".

I wonder that comment was before the news went out, like https://www.protocol.com/bulletins/microsoft-azure-arm-chips-ampere --- With Microsoft building their own arm based servers for azure, I believe that in 1-2 years we will likely see runners on arm, as being cheaper.

Another case of never say never? ;)

bwoodsend · 2022-06-06T18:39:29Z

Despite it having the same name and instruction set, Apple M1 isn't really arm64. It's a propriety chip requiring custom drivers. You can't just grab any arm64 device and run macOS on it, nor can you run Linux or Windows on an M1 machine (well Linux has just about managed it but not without a lot of work).

the fix in actions/python-versions#114 only applies to >= 3.9.1

alex · 2022-06-18T16:08:34Z

Is it expected that this might break anything in users' builds? We (pyca/cryptography) are seeing builds failing with 3.10.5 (the first version to be built with this patch as far as I can tell). here's what the failures look like: https://github.com/pyca/cryptography/runs/6948897569?check_suite_focus=true

And here's a build demonstrating that 3.10.4 is clean: pyca/cryptography#7344

The big difference one see is in the CLI args that are passed to the compiler when building C extensions. Here's what a diff of those looks like between passing and a failing build:

--- old	2022-06-18 11:24:34.000000000 -0400
+++ new	2022-06-18 11:24:43.000000000 -0400
@@ -7,6 +7,12 @@
 -fwrapv
 -O3
 -Wall
+-arch
+arm64
+-arch
+x86_64
+-isysroot
+/Applications/Xcode_12.4.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.1.sdk
 -I/usr/local/opt/sqlite/include
 -I/usr/local/opt/sqlite/include
 -I/Users/runner/openssl-macos-x86-64/include
@@ -19,12 +25,11 @@
 -march=core2
 -DUSE_OSRANDOM_RNG_FOR_TESTING
 -I/Users/runner/work/cryptography/cryptography/.tox/py310/include
--I/Users/runner/hostedtoolcache/Python/3.10.4/x64/include/python3.10
+-I/Users/runner/hostedtoolcache/Python/3.10.5/x64/include/python3.10
 -c
-build/temp.macosx-10.15-x86_64-cpython-310/_openssl.c
+build/temp.macosx-10.15-universal2-cpython-310/_openssl.c
 -o
-build/temp.macosx-10.15-x86_64-cpython-310/build/temp.macosx-10.15-x86_64-cpython-310/_openssl.o
+build/temp.macosx-10.15-universal2-cpython-310/build/temp.macosx-10.15-universal2-cpython-310/_openssl.o
 -Wconversion
 -Wno-error=sign-conversion

bwoodsend · 2022-06-18T21:44:37Z

I think that the only expected failures are if you're trying to link against single arch binaries (possibly installed via brew) or if you land with an XCode < 12.2. Neither of those look like the problem you're facing.

alex · 2022-06-19T00:49:30Z

We are actually trying to link against a single arch binary -- the OpenSSL we're linking there is x86_64 only. However, even if we switch to linking against a universal2 OpenSSL (see pyca/cryptography#7346) the issue with the sysroot exists.

Is it possible these builds aren't compatible with the macos-12 images we're using?

bwoodsend · 2022-06-19T08:31:40Z

I think that it's more likely just to be a bug either in XCode or in one of the layers of build tools if it's pointing to non-existant XCode SDKs. I'll fire up my not so beloved macOS and see if I can reproduce locally.

alex · 2022-06-19T12:31:13Z

I've put together a minimal reproducer that demonstrates that universal2 + macos-12 images aren't working for some usecases: https://github.com/alex/temp-gha-macos-python/actions/runs/2523945774

alex · 2022-06-20T23:29:31Z

I think 3.10.4 must have gotten rebuilt as universal2 in the last day or so, as it's now failing the same way 3.10.5 does.

achals · 2022-06-24T00:04:06Z

Not sure if I misunderstood, but won't this break building wheels for python projects that also compile and link to native code?

To provide context, I'm a maintainer for feast-dev/feast. We have some latency sensitive code that's been ported to go and we call into it using gopy. Up until the last release we attempted, we'd correctly produce wheels with the go libraries embedded for x86_64, and other platforms needed to install feast from source. This change ended up breaking our wheel building process and required us to do some digging to figure out what had changed.

I'd love to know if there's a better way for us to be building wheels or setting up our toolchain.

bwoodsend · 2022-06-24T05:36:24Z

Yeah, I'm having second thoughts about not making this optional. I believe that the status quo in PyPA land is very much universal2 or nothing - compiling all dependencies from source if that's what it takes.

You can opt-out of universal2 mode using the hidden set of environment variables that cibuildwheel uses. Namely, set ARCHFLAGS="-arch x86_64" _PYTHON_HOST_PLATFORM=macosx-10.9-x86_64. (In fact, now that I see that code I realise that there was a hidden way to build universal2 wheels with x86_64 Python after all. 🙄 ) Note that that 10.9 is your minimum supported macOS version. You can adjust it freely if you want to.

alex · 2022-06-24T11:54:42Z

Explicitly providing ARCHFLAGS and _PYPTHON_HOST_PLATFORM does not resolve the issue that on macOS-12 the sysroot points at a non-existant path and thus extension modules cannot be built.

bwoodsend · 2022-06-24T18:16:11Z

Can you share a build URL for that?

alex · 2022-06-24T18:19:05Z

pyca/cryptography#7346 (it's going to take a minute for the macOS 3.10 job to fail, I previously reverted the commit so the CI needs to re-run)

Edit: Specific job https://github.com/pyca/cryptography/runs/7046642554?check_suite_focus=true

hynek · 2022-06-29T10:28:07Z

Hi, this change has also broken the build process of a C extension of mine, because now CONFIGURE_CFLAGS hardcodes cross-compiling x64 and arm. That fails because I need to use different C files on each platform. Here's more details – ~~so far I haven't found a useful way to circumvent this forced cross-compilation~~. I'm not even building wheels (my cibuildwheel job seems to be working FWIW), it's just a pip install -e . (but python -m build . failed too).

In my case setting setting ARCHFLAGS helped, but it send me on an hours-long goose hunt.

bwoodsend · 2022-06-29T14:11:54Z

@miketimofeev Would you be OK with this change becoming optional and opt-in. i.e. Anyone wanting universal2 would have to use?:

- uses: setup-python@v3
  with:
    macos_arch: universal2

It feels like the safest option to me.

ssbarnea · 2022-06-29T14:20:26Z

I know that this might be inconvenient for some projects that failed to adopt the universal2, but I am against not making the default universal2 as that is what Apple recommends.

The opt-in should be about not avoiding universal2.

alex · 2022-06-29T14:31:04Z

If universal2 is going to be the default, the fact that it does not work on GHA's macOS 12 needs to be resolved. Until that happens, I encourage reverting the default.

miketimofeev · 2022-06-29T15:41:19Z

@marko-zivic-93 ^

vsafonkin · 2022-06-30T13:13:17Z

@bwoodsend looks like it will be better to disable universal2 builds at the moment and revert these changes.

cielavenir · 2022-07-01T01:59:38Z

oh reverting that change will break https://github.com/cielavenir/python-slz/actions/workflows/wheel.yaml

misl6 · 2022-07-23T09:48:27Z

I know that this might be inconvenient for some projects that failed to adopt the universal2, but I am against not making the default universal2 as that is what Apple recommends.

The opt-in should be about not avoiding universal2.

Absolutely agree.

PS: Having multiple builds (x86_64, arm64 and universal2) at the first stage, and then later switching to universal2 as a default would also work well.

albertosottile · 2022-08-15T14:41:17Z

I understand why you wanted to disable the default universal2 build for all the GHA projects, but the need described in actions/runner-images#4133 is still valid. Is there a plan to reintroduce a universal2 Python build in GHA, at least optionally?

bwoodsend mentioned this pull request Sep 23, 2021

Add universal2 builds of Python to actions/setup-python actions/runner-images#4133

Closed

10 tasks

miketimofeev mentioned this pull request Oct 18, 2021

Add Python 3.10 (release) actions/runner-images#4226

Closed

10 tasks

nikita-bykov mentioned this pull request Nov 19, 2021

Add Universal2 versions of python for use with PyInstaller actions/setup-python#271

Closed

bwoodsend commented Nov 25, 2021

View reviewed changes

TonyXiang8787 mentioned this pull request Jan 28, 2022

[FEATURE] Universal wheel file for macOS PowerGridModel/power-grid-model#13

Closed

TonyXiang8787 reviewed Jan 28, 2022

View reviewed changes

Compile in universal2 mode for macOS Python >= 3.9.1.

3c5fa6d

Using `universal2` Python environments is currently the only way to build Python wheels which support macOS M1.

bwoodsend force-pushed the universal2 branch from ac7c4e9 to 3c5fa6d Compare May 15, 2022 18:05

bwoodsend requested a review from a team May 15, 2022 18:05

vsafonkin approved these changes Jun 1, 2022

View reviewed changes

marko-zivic-93 approved these changes Jun 1, 2022

View reviewed changes

IvanZosimov approved these changes Jun 2, 2022

View reviewed changes

dmitry-shibanov approved these changes Jun 2, 2022

View reviewed changes

vsafonkin merged commit 52b4d85 into actions:main Jun 2, 2022

bwoodsend mentioned this pull request Jun 2, 2022

PyInstaller and Apple Silicon? pyinstaller/pyinstaller#5315

Closed

panticmilos mentioned this pull request Jun 3, 2022

Is there a plan to support armhf (aka armv7l) and arm64? actions/setup-python#108

Open

7 tasks

kernel-dev mentioned this pull request Jun 3, 2022

Create a universal2 binary for macOS machawk1/wail#513

Closed

bwoodsend deleted the universal2 branch June 6, 2022 18:39

charles-cooper added a commit to kelvinfan001/vyper that referenced this pull request Jun 17, 2022

bump python to 3.10

01e3616

the fix in actions/python-versions#114 only applies to >= 3.9.1

charles-cooper mentioned this pull request Jun 18, 2022

_struct.cpython-310-darwin.so is not a fat binary actions/setup-python#439

Closed

5 tasks

vsafonkin mentioned this pull request Jun 30, 2022

Disable universal2 macOS builds #175

Merged

cielavenir mentioned this pull request Jul 1, 2022

having versions-manifest.json MANIFEST_REPO_BRANCH configurable actions/setup-python#449

Closed

felixwang9817 mentioned this pull request Jul 1, 2022

Go feature server does not compile on python 3.10 on Mac OS X feast-dev/feast#2881

Closed

psobot mentioned this pull request Jul 10, 2022

Fix master build failures due to universal2 issues on macos-latest. spotify/pedalboard#113

Merged

darodi mentioned this pull request Oct 6, 2022

Kindle Comic Converter - Native Apple Silicon darodi/kcc#4

Open

ahartikainen mentioned this pull request Oct 9, 2022

ci: macos-11 image uses 10.9 wheels stan-dev/pystan#369

Closed

mattip mentioned this pull request Nov 22, 2022

Provide x64, arm64, and universal2 arch selectors for darwin actions/setup-python#547

Open

jmarshall mentioned this pull request Jul 27, 2023

Fix multiarch macOS build by removing brewed liblzma pysam-developers/pysam#1207

Merged

Compile in universal2 mode for macOS Python >= 3.9.1. #114

Compile in universal2 mode for macOS Python >= 3.9.1. #114

Conversation

bwoodsend commented Sep 23, 2021

bwoodsend commented Oct 1, 2021

miketimofeev commented Oct 1, 2021

astrofrog commented Oct 5, 2021

miketimofeev commented Oct 18, 2021 • edited Loading

bwoodsend commented Oct 18, 2021

bwoodsend Nov 25, 2021

Choose a reason for hiding this comment

TonyXiang8787 Jan 28, 2022

Choose a reason for hiding this comment

bwoodsend Jan 28, 2022

Choose a reason for hiding this comment

TonyXiang8787 Jan 28, 2022 • edited Loading

Choose a reason for hiding this comment

bwoodsend Jan 28, 2022

Choose a reason for hiding this comment

bwoodsend commented May 15, 2022

misl6 commented May 31, 2022

bwoodsend commented Jun 1, 2022 • edited Loading

bwoodsend commented Jun 1, 2022

ssbarnea commented Jun 6, 2022

bwoodsend commented Jun 6, 2022

alex commented Jun 18, 2022 • edited Loading

bwoodsend commented Jun 18, 2022

alex commented Jun 19, 2022

bwoodsend commented Jun 19, 2022

alex commented Jun 19, 2022

alex commented Jun 20, 2022

achals commented Jun 24, 2022 • edited Loading

bwoodsend commented Jun 24, 2022

alex commented Jun 24, 2022

bwoodsend commented Jun 24, 2022

alex commented Jun 24, 2022 • edited Loading

hynek commented Jun 29, 2022 • edited Loading

bwoodsend commented Jun 29, 2022

ssbarnea commented Jun 29, 2022

alex commented Jun 29, 2022

miketimofeev commented Jun 29, 2022

vsafonkin commented Jun 30, 2022 • edited Loading

cielavenir commented Jul 1, 2022

misl6 commented Jul 23, 2022

albertosottile commented Aug 15, 2022

miketimofeev commented Oct 18, 2021 •

edited

Loading

TonyXiang8787 Jan 28, 2022 •

edited

Loading

bwoodsend commented Jun 1, 2022 •

edited

Loading

alex commented Jun 18, 2022 •

edited

Loading

achals commented Jun 24, 2022 •

edited

Loading

alex commented Jun 24, 2022 •

edited

Loading

hynek commented Jun 29, 2022 •

edited

Loading

vsafonkin commented Jun 30, 2022 •

edited

Loading