Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bad CPU Type on Apple Silicon #5640

Closed
5 of 6 tasks
txoof opened this issue Mar 16, 2021 · 14 comments · Fixed by #5642
Closed
5 of 6 tasks

Bad CPU Type on Apple Silicon #5640

txoof opened this issue Mar 16, 2021 · 14 comments · Fixed by #5642
Labels

Comments

@txoof
Copy link

txoof commented Mar 16, 2021

Description of the issue

PyInstaller reports Bad CPU type in executable when run on Apple M1 Silicon (Big Sur) at various points during building and fails to build executable.

Context information (for bug reports)

  • Output of pyinstaller --version: 4.2
  • Version of Python: e.g. Python 3.9.2
  • Platform: MacOS Big Sur V 11.1 (20C69) M1 2020 Silicon
  • Same command works with on Catalina, Intel Silicon on several other machines with python 3.9.2 and pyinstaller 4.2
  • Results from development version 5.0.dev0: Same results
  • "If Things Go Wrong" Guide: does not appear to apply to this situation

Make sure everything is packaged correctly

  • start with clean installation
  • use the latest development version
  • Run your frozen program from a command window (shell) — instead of double-clicking on it
  • Package your program in --onedir mode
  • Package without UPX, say: use the option --noupx or set upx=False in your .spec-file
  • Repackage you application in verbose/debug mode. For this, pass the option --debug to pyi-makespec or pyinstaller or use EXE(..., debug=1, ...) in your .spec file.
    • I am unsure how to do this

A minimal example program which shows the error

print('hello world!')

Stacktrace / full error message of minimal example:

A minimal example does not fail to build, but does generate errors

$ pipenv run pyinstaller --onefile --noconfirm --clean --add-data Help.md:. --add-data logging_cfg.ini:. --add-data createFolders.ini:. --exclude-module iPython --noupx hello.py
18 INFO: PyInstaller: 5.0.dev0
19 INFO: Python: 3.9.2
24 INFO: Platform: macOS-11.1-arm64-arm-64bit
24 INFO: wrote /Users/aciuffo/Documents/src/portfolioCreator/create_folders/hello.spec
26 INFO: UPX is not available.
26 INFO: Removing temporary files and cleaning cache in /Users/aciuffo/Library/Application Support/pyinstaller
29 INFO: Extending PYTHONPATH with paths
['/Users/aciuffo/Documents/src/portfolioCreator',
 '/Users/aciuffo/Documents/src/portfolioCreator/create_folders']
31 INFO: checking Analysis
31 INFO: Building Analysis because Analysis-00.toc is non existent
31 INFO: Initializing module dependency graph...
32 INFO: Caching module graph hooks...
34 WARNING: Several hooks defined for module 'win32ctypes.core'. Please take care they do not conflict.
35 INFO: Analyzing base_library.zip ...
arch: posix_spawnp: /Users/aciuffo/.local/share/virtualenvs/portfolioCreator-ahiuVo4K/bin/python: Bad CPU type in executable
arch: posix_spawnp: /Users/aciuffo/.local/share/virtualenvs/portfolioCreator-ahiuVo4K/bin/python: Bad CPU type in executable
1056 INFO: Processing pre-find module path hook distutils from '/Users/aciuffo/.local/share/virtualenvs/portfolioCreator-ahiuVo4K/lib/python3.9/site-packages/PyInstaller/hooks/pre_find_module_path/hook-distutils.py'.
1056 INFO: distutils: retargeting to non-venv dir '/opt/homebrew/Cellar/python@3.9/3.9.2_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9'
2230 INFO: Caching module dependency graph...
2272 INFO: running Analysis Analysis-00.toc
2275 INFO: Analyzing /Users/aciuffo/Documents/src/portfolioCreator/create_folders/hello.py
2276 INFO: Processing module hooks...
2276 INFO: Loading module hook 'hook-xml.etree.cElementTree.py' from '/Users/aciuffo/.local/share/virtualenvs/portfolioCreator-ahiuVo4K/lib/python3.9/site-packages/PyInstaller/hooks'...
2276 INFO: Loading module hook 'hook-lib2to3.py' from '/Users/aciuffo/.local/share/virtualenvs/portfolioCreator-ahiuVo4K/lib/python3.9/site-packages/PyInstaller/hooks'...
2282 INFO: Loading module hook 'hook-_tkinter.py' from '/Users/aciuffo/.local/share/virtualenvs/portfolioCreator-ahiuVo4K/lib/python3.9/site-packages/PyInstaller/hooks'...
arch: posix_spawnp: /Users/aciuffo/.local/share/virtualenvs/portfolioCreator-ahiuVo4K/bin/python: Bad CPU type in executable
arch: posix_spawnp: /Users/aciuffo/.local/share/virtualenvs/portfolioCreator-ahiuVo4K/bin/python: Bad CPU type in executable
2289 ERROR: Tcl/Tk improperly installed on this system.
2289 INFO: Loading module hook 'hook-encodings.py' from '/Users/aciuffo/.local/share/virtualenvs/portfolioCreator-ahiuVo4K/lib/python3.9/site-packages/PyInstaller/hooks'...
arch: posix_spawnp: /Users/aciuffo/.local/share/virtualenvs/portfolioCreator-ahiuVo4K/bin/python: Bad CPU type in executable
2292 INFO: Loading module hook 'hook-distutils.util.py' from '/Users/aciuffo/.local/share/virtualenvs/portfolioCreator-ahiuVo4K/lib/python3.9/site-packages/PyInstaller/hooks'...
2293 INFO: Excluding import of lib2to3.refactor from module distutils.util
2293 INFO: Loading module hook 'hook-pickle.py' from '/Users/aciuffo/.local/share/virtualenvs/portfolioCreator-ahiuVo4K/lib/python3.9/site-packages/PyInstaller/hooks'...
2293 INFO: Excluding import of argparse from module pickle
2293 INFO: Loading module hook 'hook-heapq.py' from '/Users/aciuffo/.local/share/virtualenvs/portfolioCreator-ahiuVo4K/lib/python3.9/site-packages/PyInstaller/hooks'...
2294 INFO: Excluding import of doctest from module heapq
2294 INFO: Loading module hook 'hook-difflib.py' from '/Users/aciuffo/.local/share/virtualenvs/portfolioCreator-ahiuVo4K/lib/python3.9/site-packages/PyInstaller/hooks'...
2294 INFO: Excluding import of doctest from module difflib
2294 INFO: Loading module hook 'hook-multiprocessing.util.py' from '/Users/aciuffo/.local/share/virtualenvs/portfolioCreator-ahiuVo4K/lib/python3.9/site-packages/PyInstaller/hooks'...
2294 INFO: Excluding import of test from module multiprocessing.util
2294 INFO: Excluding import of test.support from module multiprocessing.util
2294 INFO: Loading module hook 'hook-sysconfig.py' from '/Users/aciuffo/.local/share/virtualenvs/portfolioCreator-ahiuVo4K/lib/python3.9/site-packages/PyInstaller/hooks'...
2299 INFO: Loading module hook 'hook-xml.py' from '/Users/aciuffo/.local/share/virtualenvs/portfolioCreator-ahiuVo4K/lib/python3.9/site-packages/PyInstaller/hooks'...
2321 INFO: Loading module hook 'hook-distutils.py' from '/Users/aciuffo/.local/share/virtualenvs/portfolioCreator-ahiuVo4K/lib/python3.9/site-packages/PyInstaller/hooks'...
2327 INFO: Looking for ctypes DLLs
2340 INFO: Analyzing run-time hooks ...
2342 INFO: Including run-time hook '/Users/aciuffo/.local/share/virtualenvs/portfolioCreator-ahiuVo4K/lib/python3.9/site-packages/PyInstaller/hooks/rthooks/pyi_rth_multiprocessing.py'
2344 INFO: Looking for dynamic libraries
2368 INFO: Looking for eggs
2368 INFO: Using Python library /opt/homebrew/Cellar/python@3.9/3.9.2_1/Frameworks/Python.framework/Versions/3.9/Python
2369 INFO: Warnings written to /Users/aciuffo/Documents/src/portfolioCreator/create_folders/build/hello/warn-hello.txt
2378 INFO: Graph cross-reference written to /Users/aciuffo/Documents/src/portfolioCreator/create_folders/build/hello/xref-hello.html
2382 INFO: Appending 'datas' from .spec
2383 INFO: checking PYZ
2383 INFO: Building PYZ because PYZ-00.toc is non existent
2383 INFO: Building PYZ (ZlibArchive) /Users/aciuffo/Documents/src/portfolioCreator/create_folders/build/hello/PYZ-00.pyz
2586 INFO: Building PYZ (ZlibArchive) /Users/aciuffo/Documents/src/portfolioCreator/create_folders/build/hello/PYZ-00.pyz completed successfully.
2589 INFO: checking PKG
2589 INFO: Building PKG because PKG-00.toc is non existent
2589 INFO: Building PKG (CArchive) PKG-00.pkg
3847 INFO: Building PKG (CArchive) PKG-00.pkg completed successfully.
3848 INFO: Bootloader /Users/aciuffo/.local/share/virtualenvs/portfolioCreator-ahiuVo4K/lib/python3.9/site-packages/PyInstaller/bootloader/Darwin-64bit-arm/run
3848 INFO: checking EXE
3848 INFO: Building EXE because EXE-00.toc is non existent
3848 INFO: Building EXE from EXE-00.toc
3848 INFO: Appending archive to EXE /Users/aciuffo/Documents/src/portfolioCreator/create_folders/dist/hello
3852 INFO: Fixing EXE for code signing /Users/aciuffo/Documents/src/portfolioCreator/create_folders/dist/hello
3854 INFO: Building EXE from EXE-00.toc completed successfully.

Output of executable:

$ ./dist/hello                                                      (master) ✗
[39586] Error loading Python lib '/var/folders/sg/vqd02czj31sbb6g659kyvdjr00011b/T/_MEIAVkJlt/Python': dlopen: dlopen(/var/folders/sg/vqd02czj31sbb6g659kyvdjr00011b/T/_MEIAVkJlt/Python, 10): no suitable image found.  Did find:
	/var/folders/sg/vqd02czj31sbb6g659kyvdjr00011b/T/_MEIAVkJlt/Python: mach-o, but wrong architecture
	/private/var/folders/sg/vqd02czj31sbb6g659kyvdjr00011b/T/_MEIAVkJlt/Python: mach-o, but wrong architecture

Output when building full program:

$ pipenv run pyinstaller --onedir --noconfirm --clean --add-data Help.md:. --add-data logging_cfg.ini:. --add-data createFolders.ini:. --exclude-module iPython --noupx createFolders.py
80 INFO: PyInstaller: 5.0.dev0
80 INFO: Python: 3.9.2
92 INFO: Platform: macOS-11.1-arm64-arm-64bit
94 INFO: wrote /Users/aciuffo/Documents/src/portfolioCreator/create_folders/createFolders.spec
96 INFO: UPX is not available.
96 INFO: Removing temporary files and cleaning cache in /Users/aciuffo/Library/Application Support/pyinstaller
118 INFO: Extending PYTHONPATH with paths
['/Users/aciuffo/Documents/src/portfolioCreator',
 '/Users/aciuffo/Documents/src/portfolioCreator/create_folders']
149 INFO: checking Analysis
149 INFO: Building Analysis because Analysis-00.toc is non existent
149 INFO: Initializing module dependency graph...
150 INFO: Caching module graph hooks...
152 WARNING: Several hooks defined for module 'win32ctypes.core'. Please take care they do not conflict.
153 INFO: Analyzing base_library.zip ...
arch: posix_spawnp: /Users/aciuffo/.local/share/virtualenvs/portfolioCreator-ahiuVo4K/bin/python: Bad CPU type in executable
arch: posix_spawnp: /Users/aciuffo/.local/share/virtualenvs/portfolioCreator-ahiuVo4K/bin/python: Bad CPU type in executable
1130 INFO: Processing pre-find module path hook distutils from '/Users/aciuffo/.local/share/virtualenvs/portfolioCreator-ahiuVo4K/lib/python3.9/site-packages/PyInstaller/hooks/pre_find_module_path/hook-distutils.py'.
1131 INFO: distutils: retargeting to non-venv dir '/opt/homebrew/Cellar/python@3.9/3.9.2_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9'
2367 INFO: Caching module dependency graph...
2410 INFO: running Analysis Analysis-00.toc
2413 INFO: Analyzing /Users/aciuffo/Documents/src/portfolioCreator/create_folders/createFolders.py
4635 INFO: Processing pre-find module path hook site from '/Users/aciuffo/.local/share/virtualenvs/portfolioCreator-ahiuVo4K/lib/python3.9/site-packages/PyInstaller/hooks/pre_find_module_path/hook-site.py'.
4635 INFO: site: retargeting to fake-dir '/Users/aciuffo/.local/share/virtualenvs/portfolioCreator-ahiuVo4K/lib/python3.9/site-packages/PyInstaller/fake-modules'
4920 INFO: Processing pre-safe import module hook six.moves from '/Users/aciuffo/.local/share/virtualenvs/portfolioCreator-ahiuVo4K/lib/python3.9/site-packages/PyInstaller/hooks/pre_safe_import_module/hook-six.moves.py'.
arch: posix_spawnp: /Users/aciuffo/.local/share/virtualenvs/portfolioCreator-ahiuVo4K/bin/python: Bad CPU type in executable
pre-safe-import-module hook failed, needs fixing.
@rokm
Copy link
Member

rokm commented Mar 16, 2021

This one's caused by some of our utility code still living in simpler times where 32-bit == i386 and 64-bit == x86_64:

def __wrap_python(args, kwargs):
cmdargs = [sys.executable]
# Mac OS X supports universal binaries (binary for multiple architectures.
# We need to ensure that subprocess binaries are running for the same
# architecture as python executable.
# It is necessary to run binaries with 'arch' command.
if is_darwin:
mapping = {'32bit': '-i386', '64bit': '-x86_64'}
py_prefix = ['arch', mapping[architecture]]
# Since OS X 10.11 the environment variable DYLD_LIBRARY_PATH is no
# more inherited by child processes, so we proactively propagate
# the current value using the `-e` option of the `arch` command.
if 'DYLD_LIBRARY_PATH' in os.environ:
path = os.environ['DYLD_LIBRARY_PATH']
py_prefix += ['-e', 'DYLD_LIBRARY_PATH=%s' % path]
cmdargs = py_prefix + cmdargs
.

So we end up trying to run python with x86_64 arch, which does not work on M1 unless it happens to be universal2 version (so it breaks on homebrew python, which is always single-arch). This in turn causes other utility functions such as collect_submodules in hooks come up empty...

If you need quick fix for that, you can try forcing py_prefix in the referenced code block to

py_prefix = ['arch', '-' + platform.machine()]

@rokm rokm added the bug label Mar 16, 2021
@rokm
Copy link
Member

rokm commented Mar 16, 2021

But do note that freezing on pure arm64 does not work at the moment, unless you use the WIP branch that's trying to add support for that.

@bwoodsend
Copy link
Member

I think we probably ought to rename our bootloader directories whilst we're at it in that case. Linux is also now on aarch64, ppc64le and s390x - all of which are 64bit - with aarch64 likely to become more popular thanks to the PineBook using it.

@rokm
Copy link
Member

rokm commented Mar 16, 2021

I think we probably ought to rename our bootloader directories whilst we're at it in that case. Linux is also now on aarch64, ppc64le and s390x - all of which are 64bit - with aarch64 likely to become more popular thanks to the PineBook using it.

True, that should be cleaned up at some point as well. But it'll require changes in both bootloader's waf script and python-side code. I think currently, when compiling natively for those platforms on linux, the directories actually get named correctly. (But if you try to force arch manually, you hit a check that prevents you from using anything else than '32bit' or '64bit').

For macOS specifically, if we're going with universal2 bootloader (which still allows us to extract single-arch slices during frozen application assembly) as currently planned, Darwin-64bit as bootloader directory actually works...

@txoof
Copy link
Author

txoof commented Mar 16, 2021

Fortunately code frozen on an i386 under Catalina still runs in Big Sur on both ARM and Intel hardware. I think for the moment I'll continue to build on i386 hardware as this particular program is used widely in our organization.

Is the scope of work to repair this such that I should plan to keep an i386/catalina machine for a long time? I only ask because I'll have to fight my IT department a bit to step out of the mandatory upgrades we roll out in July.

@rokm
Copy link
Member

rokm commented Mar 16, 2021

I assume you mean x86_64 instead of i386? As far as I know, Catalina dropped support for 32-bit.

I personally think that at the moment, creating an x86_64 build is the safest way to ensure that your application will run everywhere. Because it will run on current Intel hardware, and on Apple M1 (with Rosetta translation).

If you build using arm64-only python on M1, then the frozen application will obviously run only on M1, so you'll need to create an additional x86_64 build for Intel macs. (EDIT: that is, once we fully sort out support for M1)

We're planning to enable universal2 builds of bootloader, which in theory could allow building universal2 versions of frozen applications. But the problem is, this requires all used python packages to come with universal2 libraries, otherwise the application won't run on one platform or the other (for example, cryptography package is currently providing distinct x86_64 and arm64 wheels; and homebrew python is completely single-arch anyway).

So I think what we'll do is to build universal2 bootloaders by default, then have them reduced to single-arch corresponding to current running architecture during freezing process, with an option to try forcing universal2 build (and verifying that all libraries have required arch slices). This would also allow making either universal2 frozen application, or separate single-arch builds using single universal2 python environment (again, provided all packages are in universal2 format).

@rokm
Copy link
Member

rokm commented Mar 16, 2021

So long story short, you'll probably want to keep x86_64 mac around, unless you're going to upgrade all your users to M1. And if the question was Catalina vs. Big Sur, the answer is kind of the same. If all your users will receive mandatory upgrade to Big Sur, then you can probably move your build environment to Big Sur as well. Otherwise, it might be safer to stick with Catalina unless you ensure that the versions of python packages installed in Big Sur build environments are also compatible with Catalina.

@txoof
Copy link
Author

txoof commented Mar 17, 2021

@rokm
Thank you for the very clear explanation. Your thorough and complete advice is greatly appreciated. I can clearly see my path forward and understand what I need to do.

@mbanders
Copy link

mbanders commented Jun 26, 2021

I'm using the latest pyinstaller version on a new Mac mini:

% pyinstaller --version
4.3

% python
Python 3.9.2 (default, Jun 25 2021, 15:55:33) 
[Clang 12.0.5 (clang-1205.0.22.9)] on darwin

% sw_vers 
ProductName:	macOS
ProductVersion:	11.2.2
BuildVersion:	20D80

Building with pyinstaller is successful:

% pyinstaller test.spec --clean
19 INFO: PyInstaller: 4.3
19 INFO: Python: 3.9.2
25 INFO: Platform: macOS-11.2.2-arm64-arm-64bit
27 INFO: UPX is not available.
    ...
13053 INFO: Building EXE from EXE-00.toc completed successfully.

But similar problem when I try to run the executable:

% ./dist/test 
zsh: bad CPU type in executable: ./dist/test

I'm guessing I'm missing something obvious. Am I simply running it incorrectly?

@rokm
Copy link
Member

rokm commented Jun 26, 2021

@mbanders You need to use develop branch if you want to use PyInstaller on M1.

@mbanders
Copy link

@rokm Thanks ok I now uninstalled pyinstaller 4.2, and instead git cloned and installed pyinstaller from source.

Unfortunately now building fails:

% pyinstaller --clean test.spec 
19 INFO: PyInstaller: 5.0.dev0
19 INFO: Python: 3.9.2
24 INFO: Platform: macOS-11.2.2-arm64-arm-64bit
    ...
14719 INFO: Converting EXE to target arch (arm64)
Traceback (most recent call last):
  File "/Users/mianderson/Documents/.virtualenvs/first/bin/pyinstaller", line 33, in <module>
    sys.exit(load_entry_point('pyinstaller==5.0.dev0', 'console_scripts', 'pyinstaller')())
  File "/Users/mianderson/Documents/.virtualenvs/first/lib/python3.9/site-packages/pyinstaller-5.0.dev0-py3.9.egg/PyInstaller/__main__.py", line 126, in run
    run_build(pyi_config, spec_file, **vars(args))
  File "/Users/mianderson/Documents/.virtualenvs/first/lib/python3.9/site-packages/pyinstaller-5.0.dev0-py3.9.egg/PyInstaller/__main__.py", line 65, in run_build
    PyInstaller.building.build_main.main(pyi_config, spec_file, **kwargs)
  File "/Users/mianderson/Documents/.virtualenvs/first/lib/python3.9/site-packages/pyinstaller-5.0.dev0-py3.9.egg/PyInstaller/building/build_main.py", line 758, in main
    build(specfile, kw.get('distpath'), kw.get('workpath'), kw.get('clean_build'))
  File "/Users/mianderson/Documents/.virtualenvs/first/lib/python3.9/site-packages/pyinstaller-5.0.dev0-py3.9.egg/PyInstaller/building/build_main.py", line 705, in build
    exec(code, spec_namespace)
  File "test.spec", line 29, in <module>
    exe = EXE(pyz,
  File "/Users/mianderson/Documents/.virtualenvs/first/lib/python3.9/site-packages/pyinstaller-5.0.dev0-py3.9.egg/PyInstaller/building/api.py", line 524, in __init__
    self.__postinit__()
  File "/Users/mianderson/Documents/.virtualenvs/first/lib/python3.9/site-packages/pyinstaller-5.0.dev0-py3.9.egg/PyInstaller/building/datastruct.py", line 159, in __postinit__
    self.assemble()
  File "/Users/mianderson/Documents/.virtualenvs/first/lib/python3.9/site-packages/pyinstaller-5.0.dev0-py3.9.egg/PyInstaller/building/api.py", line 708, in assemble
    osxutils.binary_to_target_arch(self.name, self.target_arch,
  File "/Users/mianderson/Documents/.virtualenvs/first/lib/python3.9/site-packages/pyinstaller-5.0.dev0-py3.9.egg/PyInstaller/utils/osx.py", line 334, in binary_to_target_arch
    assert target_arch in archs, \
AssertionError: Bootloader EXE is incompatible with target arch arm64 (has arch: x86_64)!

@bwoodsend
Copy link
Member

@mbanders You need to use develop branch if you want to use PyInstaller on M1.

And recompile the bootloader.

@rokm
Copy link
Member

rokm commented Jun 26, 2021

Right, I forgot to mention that you need to rebuild the bootloader yourself.

@mbanders
Copy link

@bwoodsend @rokm

Thanks! Rebuilding the bootloader led to success.

For any future reader here:

git clone git@github.com:pyinstaller/pyinstaller.git
cd pyinstaller/bootloader/
python ./waf all
cd ..
python setup.py install

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Nov 16, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants