macOS binaries crash if codesigned with hardened runtime enabled

[micahflee]

+++ ONLY TEXT +++ DO NOT POST IMAGES +++

Description of the issue

Apple now requires all apps to get signed by their notary service before they can get run in macOS Catalina. The notary service ensures that every binary in the app has security features enabled, including that they're codesigned and that they have the Hardened Runtime capability enabled. As of January 2020, Apple's notary service won't sign any apps that don't have Hardened Runtime: https://developer.apple.com/news/?id=09032019a

I can successfully build an app bundle with PyInstaller that executes without codesigning it. I can also codesign it (but without enabled Hardened Runtime), and the app runs fine:

codesign --deep -s $DEVELOPER_ID_APPLICATION MyApp.app

But when I codesign and enable hardened runtime, the app crashes.

codesign --deep -s $DEVELOPER_ID_APPLICATION -o runtime MyApp.app

This is what happens when I try running the hardened runtime version of the app:

Traceback (most recent call last):
  File "site-packages/PyInstaller/loader/pyiboot01_bootstrap.py", line 129, in <module>
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "/Users/user/.local/share/virtualenvs/flock-agent-U4EBHnB6/lib/python3.7/site-packages/PyInstaller/loader/pyimod03_importers.py", line 623, in exec_module
    exec(bytecode, module.__dict__)
  File "ctypes/__init__.py", line 551, in <module>
  File "ctypes/__init__.py", line 273, in _reset_cache
MemoryError
[6159] Failed to execute script pyiboot01_bootstrap

After doing some searching, it appears that this project had the exact same issue, and choose to not use hardened runtime: metabrainz/picard#1340 (comment)

However, Mac binaries built after January 2020 now require this if they're going to run in macOS Catalina, so not using hardened runtime isn't an option.

Context information (for bug reports)

• Output of pyinstaller --version: 3.6
• Version of Python: 3.7.6
• Platform: macOS 10.14.6 (Mojave)
• Did you also try this on another platform? Does it work there? N/A

Stacktrace / full error message

As shown above:

Traceback (most recent call last):
  File "site-packages/PyInstaller/loader/pyiboot01_bootstrap.py", line 129, in <module>
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "/Users/user/.local/share/virtualenvs/flock-agent-U4EBHnB6/lib/python3.7/site-packages/PyInstaller/loader/pyimod03_importers.py", line 623, in exec_module
    exec(bytecode, module.__dict__)
  File "ctypes/__init__.py", line 551, in <module>
  File "ctypes/__init__.py", line 273, in _reset_cache
MemoryError
[6159] Failed to execute script pyiboot01_bootstrap

Please also see https://github.com/pyinstaller/pyinstaller/wiki/How-to-Report-Bugs
for more about what would use to solve the issue.

[micahflee]

Solved it!

The solution is to enabled hardened runtime, but give the app specific entitlements that are required for binaries built with PyInstaller. To do this, make an entitlements.plist file that contains this:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
	<!-- These are required for binaries built by PyInstaller -->
	<key>com.apple.security.cs.allow-jit</key>
	<true/>
	<key>com.apple.security.cs.allow-unsigned-executable-memory</key>
	<true/>
</dict>
</plist>

And when you codesign the app, give it these entitlements:

codesign --deep -s $DEVELOPER_ID_APPLICATION --entitlements path/to/entitlements.plist -o runtime MyApp.app

[joshbonnett]

Solved it!

The solution is to enabled hardened runtime, but give the app specific entitlements that are required for binaries built with PyInstaller. To do this, make an entitlements.plist file that contains this:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
	<!-- These are required for binaries built by PyInstaller -->
	<key>com.apple.security.cs.allow-jit</key>
	<true/>
	<key>com.apple.security.cs.allow-unsigned-executable-memory</key>
	<true/>
</dict>
</plist>

And when you codesign the app, give it these entitlements:

codesign --deep -s $DEVELOPER_ID_APPLICATION --entitlements path/to/entitlements.plist -o runtime MyApp.app

This solution solved the issue for me to. Thanks heaps micahflee!

[micahflee]

It turns out com.apple.security.cs.allow-jit isn't even needed. So just this will work:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
	<key>com.apple.security.cs.allow-unsigned-executable-memory</key>
	<true/>
</dict>
</plist>

[Legorooj]

So is this issue fixed? Can it be closed?

[thorrak]

I'm still having issues with this, unfortunately. The error I'm getting in the console when I try to run a codesigned app that has the above mentioned entitlements is as follows:

Library Validation failed: Rejecting '/private/var/folders/_g/cgqmyrn95vdbdn5f1hkr_85r0000gn/T/_MEIB8nQ2C/Python' (Team ID: none, platform: no) for process 'BrewFlasher(54984)' (Team ID: RAS94LVJ7S, platform: no), reason: mapped file has no cdhash, completely unsigned? Code has to be at least ad-hoc signed.

[thorrak]

I found the solution for that - Apparently you also need to add com.apple.security.cs.disable-library-validation per electron-userland/electron-builder#3940

[Legorooj]

Solved then?

[micahflee]

Yes I think it's solved. It might be useful to document this somewhere though, since it's required now for macOS.

[Legorooj]

@micahflee you mean apple dists need to be signed - it's mandatory now? If so, this is important to know.

[cculianu]

Yeah it's required now. You can't notarize the app if it's not hardened runtime. $%#%#% Apple.

[Legorooj]

Hmm. This need some more attention. @htgoebel @bjones1

[cculianu]

Yeah and you can't distribute the app to Catalina users if it's not notarized. Grrrrr... $#%$##$%# Apple!!!

[Legorooj]

I feel you. My personal opinion is that apple's OSes are a beautiful work of art, but too restrictive. I prefer windows and linux.

[cculianu]

Yeah same. Also the consumer market for macOS is decent if you make games or other apps (mac app store) -- it's not hard to make a little revenue from their platform.. but yeah. I feel the exact same as you about it.

[micahflee]

@micahflee you mean apple dists need to be signed - it's mandatory now? If so, this is important to know.

Yup. As @cculianu said, apps that don't have hardened runtime enabled can't be notarized, and apps that aren't notarized won't run in macOS Catalina. It's definitely important for the PyInstaller project to pay attention to.

[Legorooj]

@micahflee @cculianu This is very important. Could one of you open a new issue about this? Specifically the need to sign executables. I'd do it, but I don't run any apple, and you'll have more details than me. I'll tag the issue when it's opened.

[BoboTiG]

FTR using the plist file from that comment works like a charm for Nuxeo Drive.

[jsl303]

I'm having a problem notarizing and distributing app packaged from pyinstaller to Catalina.

In my entitlements.plist, I set com.apple.security.cs.allow-jit and com.apple.security.cs.allow-unsigned-executable-memory to true.

I also tried setting com.apple.security.cs.disable-library-validation. However, if I do that, I can't even open it. It just asks me to trash it.

Here's my command for code signing.

codesign --deep --force --options runtime --entitlements entitlements.plist --sign "$DEVELOPER" "$APP.app" --timestamp

When I open, I get this.

[1276] Error loading Python lib '/var/folders/kb/ww5w7w_s45139t2n329cr3lw0000gn/T/_MEI3Jv8Ln/libpython3.7m.dylib': dlopen: dlopen(/var/folders/kb/ww5w7w_s45139t2n329cr3lw0000gn/T/_MEI3Jv8Ln/libpython3.7m.dylib, 10): no suitable image found.  Did find:
	/var/folders/kb/ww5w7w_s45139t2n329cr3lw0000gn/T/_MEI3Jv8Ln/libpython3.7m.dylib: code signature in (/var/folders/kb/ww5w7w_s45139t2n329cr3lw0000gn/T/_MEI3Jv8Ln/libpython3.7m.dylib) not valid for use in process using Library Validation: mapped file has no cdhash, completely unsigned? Code has to be at least ad-hoc signed.
	/var/folders/kb/ww5w7w_s45139t2n329cr3lw0000gn/T/_MEI3Jv8Ln/libpython3.7m.dylib: stat() failed with errno=3

Any help would be appreciated.

Thanks!

[jsl303]

I also tried to codesign libpython3.7m.dylib before running pyinstaller, but then I got "Killed: 9" when I run the app.

It works fine if I don't codesign and allow it to run from system preference > security.

I'd like to be able to distribute normally without users having to do that though.

[cculianu]

Man Apple creates such a burden on its developers. They actually cost people like me money every time they modify their increasingly restrictive security model.

I wish they would have the good sense to fund projects such as this one.