[net6][Catalyst] Testflight published app fails to start with security policy does not allow @ path expansion

[jeromelaban]

Steps to Reproduce

1. Publish a Catalyst app to TestFlight

Expected Behavior

Application starts up properly

Actual Behavior

-------------------------------------
Translated Report (Full Report Below)
-------------------------------------

Process:               Calculator.Mobile [80989]
Path:                  /Applications/Calculator.Mobile.app/Contents/MacOS/Calculator.Mobile
Identifier:            uno.platform.calculator-canary
Version:               1.2.5 (868)
App Item ID:           1617866830
Code Type:             X86-64 (Native)
Parent Process:        launchd [1]
User ID:               504

Date/Time:             2022-04-06 23:08:26.2482 -0400
OS Version:            macOS 12.3 (21E230)
Report Version:        12
Bridge OS Version:     6.4 (19P4242)
Anonymous UUID:        


Time Awake Since Boot: 1400000 seconds

System Integrity Protection: enabled

Crashed Thread:        0

Exception Type:        EXC_CRASH (SIGABRT)
Exception Codes:       0x0000000000000000, 0x0000000000000000
Exception Note:        EXC_CORPSE_NOTIFY

Termination Reason:    Namespace DYLD, Code 1 Library missing
Library not loaded: @executable_path/../MonoBundle/libSystem.IO.Compression.Native.dylib
Referenced from: /Applications/Calculator.Mobile.app/Contents/MacOS/Calculator.Mobile
Reason: tried: '/usr/lib/libSystem.IO.Compression.Native.dylib' (no such file), (security policy does not allow @ path expansion)
(terminated at launch; ignore backtrace)

Application Specific Information:
Library not loaded: @executable_path/../MonoBundle/libSystem.IO.Compression.Native.dylib
Referenced from: /Applications/Calculator.Mobile.app/Contents/MacOS/Calculator.Mobile
Reason: tried: '/usr/lib/libSystem.IO.Compression.Native.dylib' (no such file), (security policy does not allow @ path expansion)


Thread 0 Crashed:
0   dyld                          	       0x109f710de __abort_with_payload + 10
1   dyld                          	       0x109f87212 abort_with_payload_wrapper_internal + 80
2   dyld                          	       0x109f87244 abort_with_payload + 9
3   dyld                          	       0x109f3576f dyld4::halt(char const*) + 375
4   dyld                          	       0x109f315a5 dyld4::prepare(dyld4::APIs&, dyld3::MachOAnalyzer const*) + 3995
5   dyld                          	       0x109f304d4 start + 388


Thread 0 crashed with X86 Thread State (64-bit):
  rax: 0x0000000002000209  rbx: 0x0000000000000000  rcx: 0x00007ff7b86705c8  rdx: 0x00007ff7b8670a30
  rdi: 0x0000000000000006  rsi: 0x0000000000000001  rbp: 0x00007ff7b8670610  rsp: 0x00007ff7b86705c8
   r8: 0x00007ff7b8670630   r9: 0x0000000000000000  r10: 0x000000000000009e  r11: 0x0000000000000246
  r12: 0x000000000000009e  r13: 0x00007ff7b8670a30  r14: 0x0000000000000001  r15: 0x0000000000000006
  rip: 0x0000000109f710de  rfl: 0x0000000000000246  cr2: 0x0000000109f37e74
  
Logical CPU:     0
Error Code:      0x02000209 
Trap Number:     133


Binary Images:
       0x109f2b000 -        0x109f96fff dyld (*) <dd9e80de-fb3b-349b-96a4-46874ad34d11> /usr/lib/dyld

External Modification Summary:
  Calls made by other processes targeting this process:
    task_for_pid: 0
    thread_create: 0
    thread_set_state: 0
  Calls made by this process:
    task_for_pid: 0
    thread_create: 0
    thread_set_state: 0
  Calls made by all processes on this machine:
    task_for_pid: 0
    thread_create: 0
    thread_set_state: 0

VM Region Summary:
ReadOnly portion of Libraries: Total=2964K resident=0K(0%) swapped_out_or_unallocated=2964K(100%)
Writable regions: Total=9220K written=0K(0%) resident=0K(0%) swapped_out=0K(0%) unallocated=9220K(100%)

Environment

• Catalyst 15.2.303-rc.1.55
• Azure Devops agent macos-11

Build Logs

I can provide a binlog on request.

Example Project (If Possible)

I don't have a simple repro project, but a larger project, if that's useful.

[rolfbjarne]

I don't have a simple repro project, but a larger project, if that's useful.

Yes, please, anything we can use the reproduce this would be useful.

[jeromelaban]

Here's the project csproj I'm using to get there, for catalyst specifically:
https://github.com/unoplatform/calculator/blob/canaries/dev/src/Calculator.Mobile/Calculator.Mobile.csproj#L110-L144

[spouliot]

my quick guess...

Publish a Catalyst app to TestFlight

It's a released (TestFlight) app and Catalyst requires the use of (full) AOT and static builds (unless a user .framework is used).

Library not loaded: @executable_path/../MonoBundle/libSystem.IO.Compression.Native.dylib

In such case you should not have a .dylib being referenced from the main native executable.

So having a security policy error make sense - even if the exact error message is not very helpful to diagnose the issue.

[jeromelaban]

Thanks @spouliot! I searched for some AOT options for catalyst, and assuming that it's closer to Xamarin.Mac, would AOTMode=All be enough? (I'm currently trying it out, I'll report if it helps).

[spouliot]

@jeromelaban I have not closely followed lately but it should be closer to the iOS settings/options since it shares the same restrictions (like requiring full AOT).

[jeromelaban]

@rolfbjarne So from what I can see, the only configuration which enforces AOT is arm64, and in my case, it's an x64 build.

xamarin-macios/dotnet/targets/Xamarin.Shared.Sdk.targets

Lines 867 to 877 in 9c185e1

	<PropertyGroup Condition="'$(_RunAotCompiler)' == ''">
	<!-- Don't run the AOT compiler by default -->
	<_RunAotCompiler>false</_RunAotCompiler>
	<!-- We need it for device builds for mobile platforms -->
	<_RunAotCompiler Condition="'$(_SdkIsSimulator)' != 'true' And '$(_PlatformName)' != 'macOS' And '$(_PlatformName)' != 'MacCatalyst'">true</_RunAotCompiler>
	<!-- We need it if the interpreter is enabled, no matter where -->
	<_RunAotCompiler Condition="'$(MtouchInterpreter)' != ''">true</_RunAotCompiler>
	<!-- We need it for Mac Catalyst on arm64 -->
	<_RunAotCompiler Condition="'$(RuntimeIdentifier)' == 'maccatalyst-arm64'">true</_RunAotCompiler>
	<!-- We need it for iOS/tvOS simulator on arm64 -->
	<_RunAotCompiler Condition="'$(RuntimeIdentifier)' == 'iossimulator-arm64' Or '$(RuntimeIdentifier)' == 'tvossimulator-arm64'">true</_RunAotCompiler>

Are there options to build with AOT for both x64 and arm64?

[jeromelaban]

I've tried forcing _RunAotCompiler to true, but it's not helping and I'm getting the same runtime error. I'm using the maccatalyst-x64 RID.

[rolfbjarne]

I'm having some trouble reproducing the issue, but could you attach the .ipa you uploaded to Test Flight?

It would also be nice to have the exact commands to build the Mac Catalyst project from a clean checkout.

I also believe these properties might be able to work around the problem, can you try this and see if it works?

<_LibMonoLinkMode>Static</_LibMonoLinkMode>
<_LibXamarinLinkMode>Static</_LibXamarinLinkMode>

[jeromelaban]

Thanks! I've tried again, and it does not seem to help (the AOT compiler is not being run). Here's a full binlog of what I'm trying to build: 14686-20220422-build.zip

[rolfbjarne]

Thanks! I've tried again, and it does not seem to help (the AOT compiler is not being run). Here's a full binlog of what I'm trying to build: 14686-20220422-build.zip

Those two properties won't run the AOT compiler, instead they will make the build use static (.a) libraries instead of dynamic (.dylib) libraries, so this won't happen:

Library not loaded: @executable_path/../MonoBundle/libSystem.IO.Compression.Native.dylib

because the dylib won't be there (and the main executable won't have a reference to it).

If the app still fails with these properties, please attach a new crash report.

[jeromelaban]

Ah! Got it, thanks. I'll let the app through testflight and will let you know.

[emorell96]

I am working on a MAUI .NET app and on MacCatalyst I have the same issue found here. I am guessing MAUI uses this under the hood for maccatalyst support.

The thing that started causing this was when I added an Entitlements.plist file to make it work with MSAL.

The file contains this:

<?xml version="1.0" encoding="UTF-8" ?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>keychain-access-groups</key>
  <array>
    <string>$(AppIdentifierPrefix)$(CFBundleIdentifier)</string>
  </array>
<key>com.apple.security.cs.allow-dyld-environment-variables</key>
<true/>
</dict>
</plist>

The property group doing the signing:

<PropertyGroup Condition="$(TargetFramework.Contains('-mac')) and '$(Configuration)' == 'Release'">
		<!-- <RuntimeIdentifier>ios-arm64</RuntimeIdentifier> -->
			<!-- Only needed when you have a need for entitlements -->
<ProvisionType>Manual</ProvisionType>
			<EnableAssemblyILStripping>false</EnableAssemblyILStripping>
		<CodesignEntitlement>Entitlements.plist</CodesignEntitlement>
		<CodesignKey>Apple Development: Your name</CodesignKey>
		<CodesignProvision>Your provision</CodesignProvision>
	</PropertyGroup>

Using static linking got it to build and run. Thanks!

[jeromelaban]

The workaround did the trick for me as well, noting that forcing _RunAotCompiler to true has the effect of enabling IL stripping, causing all sorts of invalid IL execution errors.

I can now run a Catalyst app from TestFlight, thanks!

[rolfbjarne]

I've finally been able to reproduce the crash locally (no need to go through TestFlight).

Requirements for the crash:

1. Must have an Entitlements.plist file (which may be empty).
2. Must use a provisioning profile with Configuration = "Mac Catalyst". Using Configuration = "Mac" works just fine.

To be explicit, this crashes:

This does not crash:

Simplified test project: keychain_experiment-0b6e8a1.zip

To repro:

1. Create a new App ID / Identifier. No need for any extra capabilities.
2. Create an "Apple Development" certificate if you don't have one already.
3. Create a new "macOS App Development" provisioning profile, and select Profile Type = Mac Catalyst + the app identifier from the previous step.
4. Configure the project to use the certificate from step 2 and the provisioning profile from step 3.
5. Build & run: dotnet build *.csproj
6. Watch it crash & burn: dotnet run

[rolfbjarne]

Unfortunately I'm not quite sure what to do here.

I couldn't find any documentation from Apple (nor anywhere else) about the differences between the two types of provisioning profiles ("Mac" or "Mac Catalyst").

All in all Mac Catalyst apps are also Mac apps, so I would think that using the Mac provisioning profile should work just fine 🤷‍♂️.

The workaround might be the actual fix:

<_LibMonoLinkMode>Static</_LibMonoLinkMode>
<_LibXamarinLinkMode>Static</_LibXamarinLinkMode>

and we might decide to make this the default if this turns out to be a bigger problem (the downside of the workaround is that it might slow build times down a little bit).

For now I think we can just wait and see how important this is, and then maybe re-evaluate at a later point (say .NET 7).

[emorell96]

This failed for me when I used a Mac provisioning profile:

[rolfbjarne]

@emorell96 they both show up as "macOS" there:

If you click on it, you'll see either "Mac" or "Mac Catalyst" for the "Configuration" field; that's the value I'm referring to.

[emorell96]

@rolfbjarne yes I see that. How do you change it to macOS?

[angelru]

@rolfbjarne

I'm having trouble with this setup:

Termination Reason: DYLD namespace, missing code library 1
Library not loaded: '@executable_path/../MonoBundle/libempty-pkcs11-catalyst.dylib'
Referenced from: '/Users/USER/Desktop/"
Reason: tried: '' (no such file), (security policy does not allow @path expansion)

		<NativeReference Include="Libs/libempty-pkcs11-catalyst.dylib">
			<Kind>Dynamic</Kind>
			<SmartLink>False</SmartLink>
		</NativeReference>

[rolfbjarne]

@angelru have you tried changing your provisioning profile as described here: #14686 (comment)?

[angelru]

@rolfbjarne

By adding:

<_LibMonoLinkMode>Static</_LibMonoLinkMode>
<_LibXamarinLinkMode>Static</_LibXamarinLinkMode>

I don't get this error when running the application:

Termination Reason: DYLD namespace, missing code library 1
Library not loaded: '@executable_path/../MonoBundle/libempty-pkcs11-catalyst.dylib'
Referenced from: '/Users/USER/Desktop/"
Reason: tried: '' (no such file), (security policy does not allow @path expansion)

		<NativeReference Include="Libs/libempty-pkcs11-catalyst.dylib">
			<Kind>Dynamic</Kind>
			<SmartLink>False</SmartLink>
		</NativeReference>

I have:

Developer ID
Create a Developer ID provisioning profile to use Apple services with your Developer ID signed applications

[angelru]

https://developercommunity.visualstudio.com/t/Signed-NET-MAUI-MACCATALYST-/10178897

[rolfbjarne]

@angelru from that link it looks like you were able to figure things out?

[angelru]

@rolfbjarne

Yes, with this configuration I got it, but I don't know why this isn't in the documentation or in the visual studio mac options.

PropertyGroup Condition="'$(TargetFramework)' == 'net6.0-maccatalyst' and '$(Configuration)' == 'Release'">
<!-- for Mac Catalyst -->
<RuntimeIdentifiers>maccatalyst-x64;maccatalyst-arm64</RuntimeIdentifiers>
<ProvisionType>Manual</ProvisionType>
<EnableAssemblyILStripping>false</EnableAssemblyILStripping>
<CodesignEntitlement>Entitlements.plist</CodesignEntitlement>
<CodesignKey>CERTIFICATE</CodesignKey>
<CodesignProvision>PROFILE</CodesignProvision>
<UseHardenedRuntime>true</UseHardenedRuntime>
</PropertyGroup>

and:
`xcrun altool --notarize-app --primary-bundle-id "idapp" --username "appleid" --password "ovoz-yksu-wrwb-pgex" --file app.zip`

[cdavidyoung]

@jfversluis So where in the .csproj are this lines supposed to be added?

<_LibMonoLinkMode>Static</_LibMonoLinkMode>
<_LibXamarinLinkMode>Static</_LibXamarinLinkMode>

I tried adding them here but am getting the same error when running from TestFlight:

	<PropertyGroup Condition="$(TargetFramework.Contains('-maccatalyst')) and '$(Configuration)' == 'Release'">
		<MtouchLink>SdkOnly</MtouchLink>
                <_LibMonoLinkMode>Static</_LibMonoLinkMode>
                <_LibXamarinLinkMode>Static</_LibXamarinLinkMode>

[kyurkchyan]

I had the same issue, and indeed, the workaround worked for me.

@cdavidyoung here's is the project properties I have that might help you

<PropertyGroup Condition="'$(Configuration)|$(TargetFramework)|$(Platform)'=='Release|net7.0-maccatalyst|AnyCPU'">
        <MtouchLink>SdkOnly</MtouchLink>
        <EnableCodeSigning>True</EnableCodeSigning>
        <EnablePackageSigning>true</EnablePackageSigning>
        <CreatePackage>true</CreatePackage>
        <CodesignEntitlements>Platforms\MacCatalyst\Entitlements.plist</CodesignEntitlements>
        
        <!--https://github.com/xamarin/xamarin-macios/issues/14686-->
        <_LibMonoLinkMode>Static</_LibMonoLinkMode>
        <_LibXamarinLinkMode>Static</_LibXamarinLinkMode>
</PropertyGroup>

[cdavidyoung]

@kyurkchyan I have a few differences and am still unable to publish the MacCatalyst app:

	<CreatePackage></CreatePackage>

	<CodesignKey>Apple Distribution: Charles Young (83Q3M48Y8B)</CodesignKey>
	<CodesignProvision>CacheAll MacCatalyst</CodesignProvision>
	<PackageSigningKey>3rd Party Mac Developer Installer: Charles Young (...)</PackageSigningKey>

            <UseInterpreter>true</UseInterpreter>

UseInterpreter turns out to be important: https://github.com/dotnet/maui/issues/13019

@kyurkchyan By chance do you also publish an iOS app?

[kyurkchyan]

@cdavidyoung I am just setting up a project, so it's just an empty MAUI app and I might not be facing the same issue as you are. Perhaps, you could post your full .csproj and I would try to compare it to mine to see any obvious differences?

[cdavidyoung]

@kyurkchyan if you are just getting started you probably won't run into the same issues until you actually try and publish an iOS or MacCatalyst app to the App Store. It is a convoluted process thanks to Apple (publishing to Samsung Galaxy and Microsoft Store is a breeze by comparison) and the Maui documentation is just not clear for me, especially because I am trying to publish both an iOS and MacCatalyst version of the same app: https://github.com/dotnet/docs-maui/issues/1403.

I have published the iOS app (CacheAll) but the MacCatalyst version has not been successful so far. I don't really want to put out my whole .csproj file until I am sure it is correct! I already pointed out some differences in the MacCatalyst release configuration: https://github.com/xamarin/xamarin-macios/issues/14686#issuecomment-1531414556

[kyurkchyan]

@cdavidyoung Got it man. I totally am with you with this... I know how an eye from a side might save the day :) BTW, whenever you find the issue, do not forget to post it here!

[spouliot]

Requiring a different provisioning profile from developers is not ideal :(

Linker flags

Doing

$ file /Applications/Uno.Gallery.Mobile.app/Contents/MacOS/Uno.Gallery.Mobile`
/Applications/Uno.Gallery.Mobile.app/Contents/MacOS/Uno.Gallery.Mobile: Mach-O 64-bit executable x86_64

shows the application was build for x86_64.

Next doing

$ otool -l /Applications/Uno.Gallery.Mobile.app/Contents/MacOS/Uno.Gallery.Mobile

we see the following

...
Load command 14
          cmd LC_LOAD_DYLIB
      cmdsize 80
         name @rpath/libSkiaSharp.framework/Versions/A/libSkiaSharp (offset 24)
...
Load command 44
          cmd LC_RPATH
      cmdsize 48
         path @executable_path/../Frameworks/ (offset 12)
...

And, as expected, the .dylib is present at the computed location.

ls -l /Applications/Uno.Gallery.Mobile.app/Contents/MacOS/../Frameworks/libSkiaSharp.framework/Versions/A/libSkiaSharp
-rwxr-xr-x  1 root  wheel  6699488 17 Jun 00:47 /Applications/Uno.Gallery.Mobile.app/Contents/MacOS/../Frameworks/libSkiaSharp.framework/Versions/A/libSkiaSharp

and it match the application x86_64 architecture.

file /Applications/Uno.Gallery.Mobile.app/Contents/MacOS/../Frameworks/libSkiaSharp.framework/Versions/A/libSkiaSharp
/Applications/Uno.Gallery.Mobile.app/Contents/MacOS/../Frameworks/libSkiaSharp.framework/Versions/A/libSkiaSharp: Mach-O 64-bit dynamically linked shared library x86_64

Entitlements

security policy does not allow @ path expansion points to security, like app-sandbox and hardened runtime.

Signed entitlements includes

{
    "beta-reports-active" = 1;
    "com.apple.application-identifier" = "PD74CHS9Z5.com.nventive.uno.gallery-canary";
    "com.apple.developer.team-identifier" = PD74CHS9Z5;
    "com.apple.security.app-sandbox" = 1;
    "com.apple.security.network.client" = 1;
    "get-task-allow" = 0;
}

https://developer.apple.com/documentation/bundleresources/entitlements/com_apple_security_cs_disable-library-validation mention that

security-hardening feature prevents a program from loading frameworks, plug-ins, or libraries unless they’re either signed by Apple or signed with the same Team ID as the main executable.

Looking at the .app I see different signers:

• "Mac App Store" for the main application
• "Apple Dev-ID" for the Skia framework

@rolfbjarne is not the CodeSign tasks resigning the frameworks ? (since it can remove extra architectures from the binaries)

note: OTOH I have no clue why a different provisioning profile would work...

[spouliot]

I can't duplicate a build with different signatures, so it might be that the Mac App Store is resigning the app (the name suggests it) and not the included framework.

Still it does not make much sense (that macOS does not accept such different signature) but I could not find other cases (at least not with any useful info) for something that sounds like something common.

As for the provisioning profiles this gems comes from Quinn “The Eskimo!”

The whole maccatalyst prefix thing is an unfortunate misstep from the very early days of Mac Catalyst; I recommend that you avoid it when creating new projects.

https://developer.apple.com/forums/thread/666138