PyInstaller and Apple Silicon?

[harshithdwivedi]

Apple just unveiled their M1 chip yesterday and that got me thinking about the impact it will have on PyInstaller and the apps packaged with it.

Will this require a complete rewrite of the way PyInstaller functions or there's an alternative to it.

[bwoodsend]

I doubt it will make any difference to us. At worst, it may require a tweak or two to the bootloader but I wouldn't expect any of the Python components to change.

[harshithdwivedi]

Oh nice! That's good to hear, thanks :)

[Durgaprasad-kelkar]

I wanted to build arm64 target build on Catalina with Pyinstaller 3.6 and Python 2.7.18. XCode12 is installed on Machine. What changes would be needed to get this.

[jay0lee]

@harshithdwivedi @bwoodsend can this be re-opened? Python 3.9.1 has experimental support for Apple's Universal2 and arm64 but it doesn't seem like it works with PyInstaller. The PyInstaller executable compiles but it runs as x86_64 emulated code.

Simple test script:

import platform
print(platform.machine())

Output with Python 3.9.1 calling test script:

arm64

Output with PyInstaller 4.1 and Python 3.9.1 compiling test script to an executable:

x86_64

Reading through https://eclecticlight.co/2020/07/20/how-to-tell-intel-code-from-universal/ and I noticed that even with the Python 3.9.1 Universal2 install PyInstaller binaries still have the x86_64 CF FA ED FE 07 magic number to start the binary file. I suspect that because MacOS sees this magic number it immediately executes it with the x86_64 emulator and the embedded Python never sees anything other than a x86_64 CPU.

[bwoodsend]

Does Python have separate binaries for Apple arm64 then? In which case we probably need to follow and provide arm64 bootloaders.

[jay0lee]

No, MacOS 11.0 introduces a new Universal2 executable file type which contains both x86_64 and arm64 native code. That way the vendor can ship one file, the user can install it on their Mac device without needing to know if it's Intel or Apple based and the binary gets native performance. The new M1 chips will run older x86_64 binaries but performance will suffer because it's using emulation.

Apple also has some information on the Universal2 format:

https://developer.apple.com/documentation/xcode/building_a_universal_macos_binary

[bwoodsend]

Thanks @jay0lee, that's useful stuff.

I notice right at the bottom it says:

If an app doesn’t contain an executable binary, the system may run it under Rosetta translation as a precautionary measure to prevent potential runtime issues. For example, the system runs script-only apps under Rosetta translation. If you verified that your app runs correctly on both Apple silicon and Intel-based Mac computers, add the LSArchitecturePriority key to your app’s Info.plist file and list the arm64 architecture first.

which I take to mean that it assumes x86_64 unless the executable is arm64 or this LSArchitecturePriority says use arm64. At a wild stretch, possibly we can set this flag and leave the bootloader be. The bootloader is just a shim and shouldn't have any noticeable performance implications itself if we can persuade Big Sur that the rest of the app is arm64-safe.

More realistically though that probably won't work and we'll just have to build these universal bootloaders. This'll probably be more complicated than it should be because our build process is so ridiculous...

[rokm]

Out of curiosity, how do PyPI and pip handle this in wheels that contain shared libraries (like PyQt5, for example)? I'd be surprised if all of those immediately switched to Universal2 format, so what happens at the moment? Do you always get x86_64 version of the package? Do you get x86_64 version on macOS running on Intel, and error on arm64 version of macOS?

If there's a chance of ending up with x86_64-only shared libraries collected from python packages, then I guess we can never be sure that we're arm64-safe, and emulation is probably the only way to go...

[bwoodsend]

Looking at NumPy 1.19.5 which released yesterday, they're all still x86_64. Wheel tags (and I think extension module tags too) are based on distutils.util.get_platform(), which being standard lib, can't easily be amended retrospectively. Does distutils.util.get_platform() work out the box on Apple Silicon? If not, setuptools are in for a bit of a mess...

[bwoodsend]

I've managed to build what I think is a hybrid boot loader. It's here. Github won't let be upload .whl so, apologies, it's on Google drive. Could someone test it for me? I've only done the default bootloader, i.e. not debug or windowed mode. I've tested it on a x86_64 Big Sur but not a arm64 one.

---

For the record, here's how to build from source. It was easier to circumvent our waf script than modify it. The build commands I used are (all ran from the bootloader folder in the root of this repo):

1. Create a build folder.

mkdir -p build

2. Compile the bootloader for x86_86 into the build folder. Unfortunately, our bootloader crashes if compiled with the recommended target triplet (-target x86_64-apple-macos_10.7 in our case). Instead you must use -arch and -mmacos_version_min. I'm not sure why this happens - it doesn't happen for a hello world C code...

clang -arch x86_64 -mmacos-version-min=10.7  `ls src/*.c` -o build/run-x86_64 -lm -lz

3. Compile it again but for arm64. This requires installing a macOS11 SDK which I did by installing Xcode 12.3. Contrary to what it says on the apple docs, you have to point clang to the SDK for macOS 11 using the -isysroot option. This is unfortunate because it hard-codes the path to Xcode and a version number in the path to the sdk.

clang -isysroot /Applications/Xcode2.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.1.sdk -arch arm64 -mmacos-version-min=11.0 -o build/run-arm64 `ls src/*.c` -lz -lc

4. Glue them together and put the output directly into PyInstaller.

lipo -create -o ../PyInstaller/bootloader/Darwin-64bit/run build/run-arm64 build/run-x86_64

You can sanity check the architectures using:

$ lipo -archs ../PyInstaller/bootloader/Darwin-64bit/run
x86_64 arm64

[papr]

When using the linked wheel, I get the following unexpected error when running a frozen program on the machine that frooze the program:

[44067] Error loading Python lib '/Users/user/test_pyinstaller/dist-10.14.6-4.2.dev0/test_platform_machine/Python': dlopen: dlopen(/Users/user/test_pyinstaller/dist-10.14.6-4.2.dev0/test_platform_machine/Python, 10): no suitable image found.  Did find:
	/Users/user/test_pyinstaller/dist-10.14.6-4.2.dev0/test_platform_machine/Python: code signature invalid for '/Users/user/test_pyinstaller/dist-10.14.6-4.2.dev0/test_platform_machine/Python'

I do not get such an error when using the current 4.2 release from pypi.

I used this command to freeze the program:

pyinstaller \
    -y \
    --clean \
    --log-level ERROR \
    --distpath "dist-$(sw_vers -productVersion)-$(pyinstaller -v)" \
    --onedir \
    --noupx \
    test_platform_machine.py

This is the program: #5494 (comment)

[rokm]

I do not get such an error when using the current 4.2 release from pypi.

That's because 4.2 automatically strips the invalid signature from python library.

The linked wheel was probably created before #5451 was merged. So you'll need to strip the signature yourself:

codesign --remove-signature /Users/user/test_pyinstaller/dist-10.14.6-4.2.dev0/test_platform_machine/Python

[papr]

Can confirm that removing the signature works.

[bwoodsend]

@papr Good to hear. Can you try freezing:

import platform
print(platform.machine())

And check that it says arm64 instead of x86_64?

[jay0lee]

That was my thought as well as confirming that same binary will run on an x86_64 Mac as well.

[papr]

I can confirm that freezing

import platform
print(platform.machine())

on macOS 10.14.6 and running the frozen program on macOS 11.1 on an Apple Silicon mac prints arm64.

🥳 🍾 🎉

[bwoodsend]

Right then. I believe our build script will need some surgery [groan]. And the vagrant file will need to be updated to install macOS 11 SDKs (I've no idea how to do this either).

[papr]

Would it be possible to get a more recent build of the hybrid boot loader?

I tried to build a BUNDLE() with the linked hybrid bootloader, and attempting to sign the *.app fails immediately with

*.app: main executable failed strict validation

(even after removing the Python signature)

Since this might be related to #5315 (comment), I thought I might give it try with a newer version before investigating further.

[bwoodsend]

Here you go. It's the 4.2 release but with the hybrid bootloader. Not entirely sure it'll fix your problem.

[papr]

@bwoodsend Then it does not contain #5451, correct? I will give it a try, nonetheless, thank you!

[jay0lee]

4.2 contained fix #5451 as called out in the changelog: https://pyinstaller.readthedocs.io/en/v4.2/CHANGES.html

…

On Mon, Jan 25, 2021, 7:42 AM Pablo Prietz ***@***.***> wrote: @bwoodsend <https://github.com/bwoodsend> Then it does not contain #5451 <#5451>, correct? I will give it a try, nonetheless, thank you! — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#5315 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABDIZMCHUG2KTB3E5PSYF3LS3VRLNANCNFSM4TRUQFEQ> .

[papr]

@jay0lee thank you for the note!
@bwoodsend It looks like this has not solved the issue. Also, it looks like this issue has nothing todo with my application being a BUNDLE(). I can reproduce the issue with the frozen

import platform
print(platform.machine())

• Frozen binaries (PyInstaller 4.2 from pypi) sign as expected
• Frozen binaries (PyInstaller from provided wheels) fail to sign with

Freezing and signing command details

$ pyinstaller \
    -y \
    --clean \
    --log-level ERROR \
    --distpath "dist-$(sw_vers -productVersion)-$(pyinstaller -v)" \
    --onedir \
    --noupx \
    test_platform_machine.py

$ codesign \ 
    --all-architectures \ 
    --force  \ 
    --verify \ 
    --verbose \ 
    -s "$sign" \ 
    --deep \ 
    dist-10.14.6-4.2/test_platform_machine/test_platform_machine

dist-10.14.6-4.2/test_platform_machine/test_platform_machine: main executable failed strict validation

Freezing and signing OS: macOS 10.14.6

$ lipo -archs dist-10.14.6-4.2/test_platform_machine/test_platform_machine
x86_64 arm64

[rokm]

As for the wheels (provided we will try to get them out again with the next release), I suppose even if we fumble the tags for macOS and they end up undiscoverable on arm64, pip will fall back to building from sdist, which will contain prebuilt universal2 bootloaders, and all will be well.

[bwoodsend]

which will contain prebuilt universal2 bootloaders, and all will be well.

Not be default they won't - unless we add them to the MANIFEST.in (which would probably be a good idea anyway - at least until those above pip and wheel versions become standard).

[rokm]

Ohhhh, right, only Windows ones are in there at the moment...

[bwoodsend]

It'll take a bit of catching up for any packages you use to switch to universal2 too (assuming they ever do - I'd be surprised if something as huge as tensorflow are willing to double that size by using fat binaries).

[rokm]

At any rate, maybe this should go into separate issue, as this one will be closed once I merge the PR.

[Safihre]

For reference:
If anyone, like us, is interested in in supporting both M1 and also macOS 10.9+.
We have achieved this in our GitHub workflow as follows:

1. Use macos-11 and set MACOSX_DEPLOYMENT_TARGET and CFLAGS.
2. Get and install Python 3.10 from python.org, not the GA one as it is not build to support 10.9+.
3. Install Python requirements of application, specifically forcing recompiling of modules to universal2.
4. Force recompiling of bootloaders from PyInstaller, which will pick up the previously set flags. Automatically checks out the latest tag from the PyInstaller repo here so we don't use some develop version.
5. Make sure to supply target_arch="universal2" to EXE() of PyInstaller.
6. Profit!

[txoof]

@Safihre I'm very interested in this solution, but I don't quite follow the steps. Would you mind writing this up a little clearer?

I have the following questions:
step 1) What needs to be done in this step?
step 2) What is GA?
step 3) I think I know what you mean, but I've never done this. A command example would be super helpful
step 5) Is that in the .spec file?

[bwoodsend]

Ooh very nice.

4. Install Python requirements of application, specifically forcing recompiling of essential modules to universal2. However, if you include some x86 packages it still works. Seems Rosetta is applied just for those x86 parts.

Why not just use:

pip wheel --no-deps --no-binary :all: pyinstaller
pip install pyinstaller*.whl

What is GA?

GA is Github Actions - Github's continuous integration platform.

However, if you include some x86 packages it still works. Seems Rosetta is applied just for those x86 parts.

I am very, very surprised by this.

[Safihre]

Why not just use:

pip wheel --no-deps --no-binary :all: pyinstaller
pip install pyinstaller*.whl

What does this do compared to my code? It includes the recompile of the bootloader? Would make it a lot easier :)

However, if you include some x86 packages it still works. Seems Rosetta is applied just for those x86 parts.

I am very, very surprised by this.

You were right, the x86 parts were not in the main workflow. Once triggered indeed they failed. Have to recompile all.

[bwoodsend]

What does this do compared to my code? It includes the recompile of the bootloader? Would make it a lot easier :)

It does exactly the same thing - it's just a bit less code...

[Safihre]

What does this do compared to my code? It includes the recompile of the bootloader? Would make it a lot easier :)

It does exactly the same thing - it's just a bit less code...

But I still need the whole copy of the source files to compile the bootloader, right? So I still need all the extra code for that?
Sorry for the follow-up questions, but reducing the code would be great indeed!

[bwoodsend]

Those extra source files are contained in the sdist (pyinstaller-4.7.tar.gz) which is what pip downloads if you specify --no-binary :all:.

[rokm]

But will it rebuild the bootloader? Considering the sdist contains the pre-built macOS (and Windows) bootloaders as well?

[bwoodsend]

Ohhh, you're right. I forgot that those bootloaders were still lurking about in there. @Safihre Ignore everything I said above.

[rokm]

What if we added a switch (controlled by environment variable) that would force rebuild of the bootloader, even if they are already present?

[Safihre]

I was hoping for something like that!

[bwoodsend]

Sounds like a good idea.

[loyio]

Any update?

[Safihre]

It's already part of PyInstaller, we happily use it!

PYINSTALLER_COMPILE_BOOTLOADER=1 MACOSX_DEPLOYMENT_TARGET=10.9 pip3 install pyinstaller --no-binary pyinstaller

[bwoodsend]

Just a FYI: I've finally got actions/python-versions#114 merged so the Python installations provided by GitHub Actions will become universal2 at some point in the near future. The deployment targets will be 10.15 though so this isn't going to help anyone supporting older macOS's than that.

[hacksick]

I am trying to convert .py to .app but it doesn't work no matter how much I try!
Am on the m1 Mac.
It works on my Mac but when I upload it to cloud and redownload it, its opens up in a notepad!

[harshithdwivedi]

You need to chmod the file to be ran as an executable

chmod a+x friendcordmac

[hacksick]

You need to chmod the file to be ran as an executable

chmod a+x friendcordmac

bro u da best, thank you so much