macOS: add support for macOS 11 and Apple M1 Silicon (arm64) #5581

rokm · 2021-02-23T12:01:26Z

This is a cleaned-up version of the macos11-arm-support branch, which aims to add proper support for macOS 11 Big Sur and Apple M1 Silicon (arm64) architecture.

Fixes #5315.

In particular:

the bootloaders are now built as unversal2 (x86_64, arm64) binaries by default. This allows native execution on arm64, and thus fixes Frozen platform.machine() returns x86_64 on Apple Silicon M1 macs #5494. As this requires a recent version of Xcode toolchain (12.2 or newer), our wafscript now has --no-universal2 option to enable building x86_64-only thin-arch bootloaders with older toolchains.
the assembly of frozen executable now (hopefully) supports both thin-arch and universal2 bootloaders, along with their peculiarities (signing requirements on arm64 and its implications, header adjustment, etc.). Fixes PyInstaller and Apple Silicon? #5315 (comment).
the previously-introduced removal of invalid signature from Python library is superseded by re-signing with dummy signature. While removal works on x86_64, arm64 has stricter signing requirements, so we need to replace the signature instead of removing it (which I believe PyInstaller and Apple Silicon? #5315 (comment) is about).
our crossosx Vagrant VM now builds the toolchain from Command Line Tools for Xcode instead of full Xcode package to save bandwidth and time, and to alleviate storage requirements.
the clang in linux64 Vagrant VM is now updated to clang-11 from apt.llvm.org so it can build universal2 bootloader.

So once the bootloaders are rebuilt using Xcode 12.2 or later, and with corresponding MacOS SDK 11.x, ctypes.util.find_library()˙ will work when used with universal2` version of python 3.9.1 (or later), which fixes #5491.

We now also need the full back-to-front scan for embedded archive (#5511; the commit is for now included in the PR for easier testing), because the old signature search code in bootloader does not work with fat binaries. (While we could extend it to handle both types, the brute force search is needed to fix other issues as well).

Note: the PR does not include rebuilt bootloaders.

BoboTiG

That is awesome @rokm 🤯
Code review seems good, I will take time to test later in the afternoon.

BoboTiG · 2021-02-23T12:20:38Z

.github/workflows/macos.yml

@@ -49,7 +49,7 @@ jobs:

          # Compile bootloader
          cd bootloader
-          python waf distclean all
+          python waf --no-universal2 distclean all


I guess we will wait for macos-11.0 public availability before being able to enable unversal builds.
As stated in the doc:

The macOS 11.0 virtual environment is currently provided as a private preview only.
The "macos-latest" YAML workflow label still uses the macOS 10.15 virtual environment.

BoboTiG · 2021-02-23T16:08:27Z

Works for me 👍

arsenetar · 2021-04-07T00:59:40Z

Just letting you know this fixed the first issue I was running into trying to build dupeguru on a M1 mac. (The binary was registering as x86_64 instead of universal or arm64 and resulting in some issue with libraries). I was using python 3.9.2 on macOS 11.2.3. The other problems I am still running into appear to be not related to these changes.

BoboTiG · 2021-04-11T18:05:20Z

PyInstaller/utils/osx.py

+        identity = '-'  # ad-hoc signing
+
+    logger.debug("Signing file %r", filename)
+    cmd_args = ['codesign', '-s', identity, '--force', '--all-architectures',


Should we specify --options runtime and --timestamp to ease notarization?
Another thing to consider, eventually, would be to add another option to allow passing an entitlements file (--entitlements path/to/entitlements.plist).

With all those parameters, one would be allowed to completely sign the app without having to post-process again all files after PyInstaller, which is the case for all code-signed softwares using it for now.

Are these options relevant to all collected binaries, or only to the generated executable (or perhaps only to .app bundle)?

To everything packaged into an app, and to the final .app bundle also.

FTR here is the post-process I am using to sign an .app: https://github.com/nuxeo/nuxeo-drive/blob/9a5827710f113ba6ff61124adbbc7cb3b58b462a/tools/osx/deploy_ci_agent.sh#L108-L133 (put aside the extension stuff, it is not revelant here).

Does hardened runtime require a real codesign identity? Trying to use it in combination with ad-hoc signing or with self-signed certificate results in error:

[2171] Error loading Python lib '/Users/rok/Development/pyi-macos11/dist/program/Python': dlopen: dlopen(/Users/rok/Development/pyi-macos11/dist/program/Python, 10): no suitable image found. Did find: /Users/rok/Development/pyi-macos11/dist/program/Python: code signature in (/Users/rok/Development/pyi-macos11/dist/program/Python) not valid for use in process using Library Validation: mapped file has no Team ID and is not a platform binary (signed with custom identity or adhoc?)

Of course, I am testing it right now.

It is taking more time than I expected. I am stuck on:

MyApp.app: code has no resources but signature indicates they must be present

I will try to isolate the thing to understand what is not good. I have custom data, maybe it does not help. Will keep you updated.

In your tests, do those commands pass?

$ codesign --verbose=4 --deep --strict file.app $ spctl --assess --verbose file.app

Hmm, I think you need to sign the .app bundle - this PR explicitly doesn't do that, so you still need to do it yourself. But you probably don't have to do --deep signing, since individual binaries are already signed.

I've now added an automatic attempt to sign the .app bundle as well. In contrast to library collection, we don't consider a failure here to be fatal, just in case some manual adjustments to bundle are needed.

rokm · 2021-04-12T11:01:17Z

This PR is now rebased on top of develop and revised according to feedback and lessons learned so far. Most notably:

we now re-sign all collected binaries, as modifying the library paths in their headers invalidates any present signatures (and arm64 slices must have at least ad-hoc signature). By default, we do ad-hoc signing, but an actual identity can be passed via new --codesign-identity CLI switch (or codesign_identity argument to EXE in spec file), which also turns on hardened runtime option.
by default, we now target current running architecture. This is because even with universal2 python installation, one might end up with single-arch package wheels. So current running architecture is the only one with the guarantee that all required libraries are available (not to mention that Homebrew python is always single-arch). Building universal2 application must therefore be explicitly enabled using new --target-arch CLI switch (or target_arch argument to EXE in spec file). This can also be used to attempt building a single arch (x86_64 or arm64) version from universal2 environment (provided all binaries are compatible with that).
we now validate all binaries for arch compatibility, to catch mismatches at build time and avoid run-time surprises.
when targeting a single arch, we also convert any fat binaries (including the bootloader) to single-arch thin binaries, to reduce the bundle size

This should enable creation of frozen applications for all possible arch combinations; single-arch with single-arch python, universal2 with universal2 python, and single-arch with universal2 python.

For example, freezing a "Hello world" script with universal2 python on x86_64 macOS:

 $ pyinstaller hello.py
 $ lipo -archs dist/hello/hello 
 x86_64
$ lipo -archs dist/hello/Python 
x86_64

Building universal2 version:

$ pyinstaller hello.py --distpath dist-universal2 --target-arch universal2
$ lipo -archs dist-universal2/hello/hello 
x86_64 arm64
$ lipo -archs dist-universal2/hello/Python 
x86_64 arm64

Building arm64-only version ("cross-compiling"):

$ pyinstaller hello.py --distpath dist-arm64 --target-arch arm64
$ lipo -archs dist-arm64/hello/hello 
arm64
$ lipo -archs dist-arm64/hello/Python 
arm64

Trying to build a universal2 or arm64 version of a progam that imports psutil, however, will fail due to psutil providing single-arch wheel:

$ pyinstaller program_psutil.py --distpath dist-universal2 --target-arch universal2
...
AssertionError: [...]/venv/lib/python3.9/site-packages/psutil/_psutil_osx.cpython-39-darwin.so is not a fat binary!

$ pyinstaller program_psutil.py --distpath dist-arm64 --target-arch arm64
...
AssertionError: [...]/venv/lib/python3.9/site-packages/psutil/_psutil_osx.cpython-39-darwin.so is incompatible with target arch arm64 (has arch: x86_64)!

BoboTiG · 2021-05-01T12:41:47Z

I'll have time to check in coming days, I did not forgot ;)

nzjrs · 2021-05-01T23:03:32Z

It might be worth explaining the differences and consequences between thin and fat binaries especially in the context of MacOS +/- universal2 builds.

Add option to specify code signing identity via codesign_identity argument to EXE(), or using --codesign-identity CLI switch, as an alternative to default ad-hoc signing. Also split the bincache in additional sub-directories based on codesign identity (sha256 of identity string if one is provided, "adhoc" otherwise) to prevent interference. This additional split applies only to macOS at the moment. This effectively allows user to sign all binaries within a onefile build with their codesign identity. NOTE: while this signs all collected binaries and the resulting executable, we do NOT sign app bundles. Those have corner cases tat need to be resolved manually, as has been the case up until now.

On macOS, we've been re-writing library paths on cached results as well. This is probably (hopefully?) unnecessary, as those path adjustments depend only on dist_nm. As path rewriting modifies headers, this forces us to re-sign the binaries on practicaly every run, which negates the point of the cache. However, the strip + upx combination should probably properly forward dist_nm to its call to checkCache (even if the paths get subsequently rewritten again).

Update the virtual machines involved in building macOS bootloaders to produce universal2 bootloaders built against MacOS SDK 11.x. Building against MacOS SDK 11.x enables proper support for Big Sur; for example, it allows ctypes.util.find_library() to find system libraries in the dlyd share cache (provided that python was also built against SDK 11.x). Universal2 bootloaders allow native execution both on x86_64 and the new Apple M1 Silicon (arm64) machines. In order to support the above, the `build-osxcross` virtual machine is updated to use MacOS 11.x SDK from Command Line Tools for Xcode 12.x. This also reduces storage requirements and speeds up the extraction process, as we do not need to use full Xcode package anymore. The `linux64` virtual machine is updated to use clang-11 from apt.llvm.org repository, as clang-7 that is provided by Debian Buster fails to build universal2 binaries.

Prevents the following error ``` /dev/vda does not exist, so cannot grub-install to it! You must correct your GRUB install devices before proceeding: dpkg-reconfigure grub-pc dpkg: error processing package grub-pc (--configure): installed grub-pc package post-installation script subprocess returned error exit status 1 Errors were encountered while processing: grub-pc E: Sub-process /usr/bin/dpkg returned an error code ``` which seems to be caused by the latest grub-pc update.

It seems the toolchain on our CI lacks support for building universal2 bootloaders.

Extend the EXE() arguments with entitlements_file argument, and add corresponding --osx-entitlements-file command-line switch. If specified, the entitlements file is passed to the codesign utility via --entitlements switch when signing the collected binaries and the generated executable.

Using the same parameters as when signing collected binaries, plus --deep in case we missed any files that ought to be signed as well.

On macOS, avoid appending -arm suffix to bootloader directory if we are on arm64 (M1) platform. So instead of Darwin-64bit-arm, we always use Darwin-64bit. Applies both to bootloader's wscript as well as frozen application assembly pipeline (or rather, PyInstaller.PLATFORM). This is because we now build universal2 bootloaders, so there is no need for two arch-specific locations. Furthermore, using -arm suffix for bootloader directory would prevent M1 users from using pre-compiled bootloaders from git tree (unless we saved one copy in each location) and in binary wheels (unless we split wheels into x86_64 and arm64 one). Overall, considering the current design of x86_64/arm64 support, it makes more sense to unify the bootloader directory for macOS.

rokm · 2021-06-01T16:24:20Z

The latest revision forces the bootloader directory to be always Darwin-64bit, as opposed to Darwin-64bit-arm on arm64 M1 (as reported in #5315 (comment)).

Keeping two distinct directories when we have a universal2 bootloader is rather pointless and presents logistic hassle (pre-built bootloaders in git tree, binary wheels for macOS).

BoboTiG · 2021-06-01T16:31:08Z

OK, I allocated 1h tomorrow to test it and unlock the merge.

BoboTiG · 2021-06-02T08:46:08Z

Excellent job @rokm, it works like a charm!
I was only able to test the architecture part of the PR, not the full codesign one as my setup requires more work (I need to unlock the keychain first, but I will not be able to try out until several days).

I would say it is OK to merge, and if we need to do adjustments on the codesign part, then it is not a big deal :)

bwoodsend · 2021-06-02T11:28:41Z

Thank @BoboTiG. @rokm go ahead and merge this when you're ready...

BoboTiG reviewed Feb 23, 2021

View reviewed changes

BoboTiG added area:bootloader Caused be or effecting the bootloader platform:OS X labels Feb 23, 2021

BoboTiG approved these changes Feb 23, 2021

View reviewed changes

bwoodsend approved these changes Feb 23, 2021

View reviewed changes

rokm mentioned this pull request Feb 25, 2021

PyInstaller and Apple Silicon? #5315

Closed

rokm mentioned this pull request Apr 5, 2021

Pyinstaller can't find dynamically linked libs for Mac OS X Big Sur Beta #5107

Closed

rokm force-pushed the macos11-arm64-support-cleanup branch 4 times, most recently from 3dfd622 to f46b856 Compare April 11, 2021 15:01

BoboTiG reviewed Apr 11, 2021

View reviewed changes

rokm force-pushed the macos11-arm64-support-cleanup branch 2 times, most recently from 92f63f5 to f46b856 Compare April 11, 2021 19:06

Legorooj force-pushed the develop branch 4 times, most recently from 3a8eccd to e20e74c Compare April 17, 2021 04:06

rokm mentioned this pull request Apr 18, 2021

Codesign MacOS Big Sur exec file validation and illegal hardware instruction errors - 2021 - Previous fixes don't work anymore - #5743

Closed

rokm force-pushed the macos11-arm64-support-cleanup branch 6 times, most recently from ec6875e to b9dd798 Compare May 1, 2021 10:34

rokm added 9 commits June 1, 2021 18:01

CI: disable universal2 bootloader building

1bd1025

It seems the toolchain on our CI lacks support for building universal2 bootloaders.

doc: document the new macOS multi-arch support and code-signing behavior

8225f4c

osxutils: sign_binary: enable hardened runtime if identity is provided

e0aca36

building: macOS: attempt to automatically sign the .app BUNDLE

776b9a5

Using the same parameters as when signing collected binaries, plus --deep in case we missed any files that ought to be signed as well.

rokm force-pushed the macos11-arm64-support-cleanup branch from 790f9e4 to b349b0b Compare June 1, 2021 16:16

rokm force-pushed the macos11-arm64-support-cleanup branch from b349b0b to aa4b8d9 Compare June 1, 2021 16:17

rokm merged commit 660a2e3 into pyinstaller:develop Jun 2, 2021

rokm mentioned this pull request Jun 2, 2021

Mac OSX: Added code signing capability with --osx-codesign-identity arg. #3991

Closed

rokm deleted the macos11-arm64-support-cleanup branch June 2, 2021 14:02

bwoodsend mentioned this pull request Jun 2, 2021

Universal 2 Wheels for Apple M1 #5879

Closed

marcelstoer mentioned this pull request Jun 8, 2021

Unexpected quitting after pressing the flash button on Apple M1 machine (Big Sur) marcelstoer/nodemcu-pyflasher#77

Closed

bwoodsend mentioned this pull request Jun 25, 2021

Problem when notarize macOS application #5942

Closed

rokm mentioned this pull request Jul 2, 2021

some macOS libraries get corrupted when packaged with pyinstaller #5965

Closed

ErikBjare mentioned this pull request Jul 5, 2021

ci: progress on codesigning macOS builds ActivityWatch/activitywatch#620

Merged

2 tasks

This was referenced Jul 19, 2021

Look into macOS signing/notarization qutebrowser/qutebrowser#5020

Open

M1/Apple Silicon version of Qutebrowser? qutebrowser/qutebrowser#6478

Open

cho-m mentioned this pull request Jul 26, 2021

pyinstaller: fix build for Linux and use thin binaries on macOS Homebrew/homebrew-core#81847

Closed

6 tasks

rokm mentioned this pull request Aug 26, 2021

PyInstaller 4.5.1 "codesign failure!" on macos #6167

Closed

rt121212121 mentioned this pull request Jun 20, 2022

PyInstaller changes to MacOS build code-signing electrumsv/electrumsv#903

Closed

github-actions bot locked as resolved and limited conversation to collaborators Nov 16, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

macOS: add support for macOS 11 and Apple M1 Silicon (arm64) #5581

macOS: add support for macOS 11 and Apple M1 Silicon (arm64) #5581

rokm commented Feb 23, 2021

BoboTiG left a comment

BoboTiG Feb 23, 2021

BoboTiG commented Feb 23, 2021

arsenetar commented Apr 7, 2021

BoboTiG Apr 11, 2021

rokm Apr 11, 2021

BoboTiG Apr 11, 2021 •

edited

BoboTiG Apr 11, 2021 •

edited

rokm Apr 11, 2021

BoboTiG Apr 13, 2021

BoboTiG Apr 13, 2021 •

edited

BoboTiG Apr 13, 2021

rokm Apr 13, 2021

rokm May 1, 2021

rokm commented Apr 12, 2021 •

edited

BoboTiG commented May 1, 2021 •

edited

nzjrs commented May 1, 2021

rokm commented Jun 1, 2021 •

edited

BoboTiG commented Jun 1, 2021

BoboTiG commented Jun 2, 2021

bwoodsend commented Jun 2, 2021

macOS: add support for macOS 11 and Apple M1 Silicon (arm64) #5581

macOS: add support for macOS 11 and Apple M1 Silicon (arm64) #5581

Conversation

rokm commented Feb 23, 2021

BoboTiG left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

BoboTiG commented Feb 23, 2021

arsenetar commented Apr 7, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

BoboTiG Apr 11, 2021 • edited

Choose a reason for hiding this comment

BoboTiG Apr 11, 2021 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

BoboTiG Apr 13, 2021 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rokm commented Apr 12, 2021 • edited

BoboTiG commented May 1, 2021 • edited

nzjrs commented May 1, 2021

rokm commented Jun 1, 2021 • edited

BoboTiG commented Jun 1, 2021

BoboTiG commented Jun 2, 2021

bwoodsend commented Jun 2, 2021

BoboTiG Apr 11, 2021 •

edited

BoboTiG Apr 11, 2021 •

edited

BoboTiG Apr 13, 2021 •

edited

rokm commented Apr 12, 2021 •

edited

BoboTiG commented May 1, 2021 •

edited

rokm commented Jun 1, 2021 •

edited