[net6][iOS] `Ran out of trampolines of type 1` running on x64 simulator #68808

rolfbjarne · 2022-05-03T11:52:41Z

From @jeromelaban on Fri, 29 Apr 2022 15:13:56 GMT

Steps to Reproduce

Build and run a large app with the x64 iOS simulator

Expected Behavior

The app runs without failing.

Actual Behavior

Ran out of trampolines of type 1 in '7B5B7D87-1D1E-43DB-9BB1-12062FA2B779/data/Containers/Bundle/Application/7599D867-453F-4D78-99CA-4EDBDB515629/My.app/System.Private.CoreLib.dll' (limit 4096)

Environment

.NET 6 Mobile RC2

Build Logs

On demand.

Other details

Note that this does not reproduce on a physical device.

It tried changing this:

<MonoAOTCompilerDefaultAotArguments Include="nimt-trampolines=4000" />
<MonoAOTCompilerDefaultAotArguments Include="ntrampolines=10000" />
<MonoAOTCompilerDefaultAotArguments Include="nrgctx-fetch-trampolines=256" />
<MonoAOTCompilerDefaultAotArguments Include="ngsharedvt-trampolines=8192" />
<MonoAOTCompilerDefaultAotArguments Include="nftnptr-arg-trampolines=8000" />
<MonoAOTCompilerDefaultAotArguments Include="nrgctx-trampolines=31000" />

But since the AOT compiler does not seem to run in this context, it's likely not using these parameters.

Related: #53033

Copied from original issue xamarin/xamarin-macios#14855

The text was updated successfully, but these errors were encountered:

rolfbjarne · 2022-05-03T11:52:42Z

From @rolfbjarne on Fri, 29 Apr 2022 15:19:55 GMT

We're going to need a way to reproduce this, would it be possible to get access to your project?

rolfbjarne · 2022-05-03T11:52:43Z

From @msftbot[bot] on Fri, 29 Apr 2022 15:20:13 GMT

Hi @jeromelaban. We have added the "need-info" label to this issue, which indicates that we have an open question for you before we can take further action. This issue will be closed automatically in 7 days if we do not hear back from you by then - please feel free to re-open it if you come back to this issue after that time.

rolfbjarne · 2022-05-03T11:52:43Z

From @jeromelaban on Fri, 29 Apr 2022 15:22:39 GMT

Yes, it's this one, from this commit: unoplatform/Uno.Gallery@233b06d.

You'll need to build Uno.Gallery/Uno.Gallery.Mobile/Uno.Gallery.Mobile.csproj for net6.0-ios. I'm doing this on windows, if that matters.

rolfbjarne · 2022-05-03T11:52:44Z

From @jeromelaban on Fri, 29 Apr 2022 15:26:55 GMT

One additional information I discovered, if I disable the interpreter, the application runs properly. The hint was in the stack trace:

	  at <unknown> <0xffffffff>
	  at System.Object:__icall_wrapper_ves_icall_mono_delegate_ctor_interp <0xffffffff>
	  at <>c__DisplayClass0_0:<Build>b__0 <0x003c2>
	  at Uno.Material.__Resources._mergedpages_c69cbc6d79c924ed978294ad194451a9_mergedpagesRDSC61:Build <0x005b8>
	  at <>c:<Get_3_289>b__126_2 <0x0001e>

rolfbjarne · 2022-05-03T11:52:45Z

From @rolfbjarne on Tue, 03 May 2022 11:52:04 GMT

I can reproduce, here's a crash report: https://gist.github.com/rolfbjarne/8038c203eec8a33fc580bee41866d75b

I believe this is because:

We enable the AOT compiler when using the interpreter, because we need to AOT-compile System.Private.CoreLib.dll.
We're not in FullAOT mode (we're interpreting), so this block isn't executed:

runtime/src/mono/mono/mini/aot-runtime.c

Lines 2278 to 2283 in c73386b

    
           if (mono_aot_only && !mono_llvm_only) { 
        
           	char *code; 
        
           	find_amodule_symbol (amodule, "specific_trampolines_page", (gpointer *)&code); 
        
           	amodule->use_page_trampolines = code != NULL; 
        
           	/*g_warning ("using page trampolines: %d", amodule->use_page_trampolines);*/ 
        
           }

Which means we'll never get here, because USE_PAGE_TRAMPOLINES will be false:

runtime/src/mono/mono/mini/aot-runtime.c

Line 5858 in c73386b

code = (guint8 *)get_new_rgctx_trampoline_from_page (addr, ctx);

And instead we'll go here:

runtime/src/mono/mono/mini/aot-runtime.c

Line 5860 in c73386b

    
           code = (guint8 *)get_numerous_trampoline (MONO_AOT_TRAMP_STATIC_RGCTX, 2, &amodule, &got_offset, NULL);

And eventually end up here:

runtime/src/mono/mono/mini/aot-runtime.c

Lines 5770 to 5771 in c73386b

    
           g_error ("Ran out of trampolines of type %d in '%s' (limit %d)\n", 
        
           		 tramp_type, image ? image->name : MONO_ASSEMBLY_CORLIB_NAME, amodule->info.num_trampolines [tramp_type]);

The current workaround is to disable the interpreter.

I'm moving to dotnet/runtime, since I believe that's where the issue is.

rolfbjarne · 2022-05-03T11:53:11Z

CC @steveisok @vargaz

vargaz · 2022-05-03T19:21:06Z

Is the x64 ios simulator config running in aot-only mode ?

vargaz · 2022-05-03T19:22:59Z

The infinite trampoline stuff is not implemented on x64, so either we need to increase the default number of rgctx trampolines we emit into the corlib aot image, or enable the user to set it.

steveisok · 2022-05-03T19:31:11Z

@rolfbjarne maybe setting the defaults higher when you aot would be the way to go?

rolfbjarne · 2022-05-04T07:39:31Z

The infinite trampoline stuff is not implemented on x64

Would this be possible? What could be the timeline be (.NET 7?)?

@rolfbjarne maybe setting the defaults higher when you aot would be the way to go?

Yes, we can either allow the user to increment these values (I didn't know we didn't have infinite trampolines on all our platforms, so it's currently not possible to increment the number of trampolines manually), or set a higher default (or do both).

That said, the final fix is to implement infinite trampolines.

rolfbjarne · 2022-05-04T07:42:18Z

Yes, we can either allow the user to increment these values (I didn't know we didn't have infinite trampolines on all our platforms, so it's currently not possible to increment the number of trampolines manually), or set a higher default (or do both).

I've filed xamarin/xamarin-macios#14887 so we can look at this in the meantime.

vargaz · 2022-05-05T01:53:05Z

The infinite trampoline code might conflict with future apple security measures, so adding it could lead to problems in the future. The question is why is this configuration which rans on a JIT capable system is using aot trampolines in the first place.

vargaz · 2022-05-05T02:05:27Z

Looks like we have a MONO_AOT_MODE_INTERP and a MONO_AOT_MODE_INTERP_ONLY configuration, the former is based on full aot so it will try to use aot trampolines, the latter doesn't.

steveisok · 2022-05-05T02:08:36Z

This mode was introduced primarily to get interp mode working on arm64 where JIT fallback isn't realistic.

/cc @lambdageek

vargaz · 2022-05-05T02:29:55Z

@rolfbjarne Would it be possible to use MONO_AOT_MODE_INTERP_ONLY instead MONO_AOT_MODE_INTERP on x64 ?
I.e. emit

#ifdef      __x86_64__
mono_jit_set_aot_mode (MONO_AOT_MODE_INTERP_ONLY)
#else
mono_jit_set_aot_mode (MONO_AOT_MODE_INTERP)
#endif

We might need to modify the ios x64 simulator runtimes to not disable the jit.

lambdageek · 2022-05-05T13:41:45Z

We need FullAOT trampolines for the interpreter on arm64 because apple doesn't allow JITing on the M1 hardware.

For the x64 Simulator we FullAOT the interpreter trampolines mostly because we can just have a single configuration for both M1 and x64.

The infinite trampoline code might conflict with future apple security measures, so adding it could lead to problems in the future. The question is why is this configuration which rans on a JIT capable system is using aot trampolines in the first place.

Do you mean on the M1 or x64? I doubt Apple would introduce additional security measures on x64 that are not already present on the M1. And if they did decide to lock down x64, I think JITing would be disallowed before page remapping.

rolfbjarne · 2022-05-11T20:19:13Z

@rolfbjarne Would it be possible to use MONO_AOT_MODE_INTERP_ONLY instead MONO_AOT_MODE_INTERP on x64 ?

I can certainly try and see if it works (I'm assuming it can work without too much effort, since it did work at some point), but the question (which I can't really answer) is: do we want to do that, or implement infinite trampolines on x86_64?

steveisok · 2022-05-11T20:29:44Z

@rolfbjarne Would it be possible to use MONO_AOT_MODE_INTERP_ONLY instead MONO_AOT_MODE_INTERP on x64 ?

I can certainly try and see if it works (I'm assuming it can work without too much effort, since it did work at some point), but the question (which I can't really answer) is: do we want to do that, or implement infinite trampolines on x86_64?

My vote: I would rather run the same configuration on both x64 and arm64.

rolfbjarne · 2022-05-11T20:33:05Z

@rolfbjarne Would it be possible to use MONO_AOT_MODE_INTERP_ONLY instead MONO_AOT_MODE_INTERP on x64 ?

I can certainly try and see if it works

It didn't, it aborts with:

error: Failed to load AOT module 'System.Private.CoreLib' while running in aot-only mode: not compiled with --aot=llvmonly.

=================================================================
	Native Crash Reporting
=================================================================
Got a SIGABRT while executing native code. This usually indicates
a fatal error in the mono runtime or one of the native libraries
used by your application.
=================================================================

=================================================================
	Native stacktrace:
=================================================================
	0x10b22fa45 - /Users/rolf/Library/Developer/CoreSimulator/Devices/D21038CB-BC03-4F58-886D-D8106F87F512/data/Containers/Bundle/Application/3F282177-5E2C-4838-987B-507586C1BC52/MyInterpretedApp.app/libmonosgen-2.0.dylib : mono_dump_native_crash_info
	0x10b1cf6ee - /Users/rolf/Library/Developer/CoreSimulator/Devices/D21038CB-BC03-4F58-886D-D8106F87F512/data/Containers/Bundle/Application/3F282177-5E2C-4838-987B-507586C1BC52/MyInterpretedApp.app/libmonosgen-2.0.dylib : mono_handle_native_crash
	0x10b22f352 - /Users/rolf/Library/Developer/CoreSimulator/Devices/D21038CB-BC03-4F58-886D-D8106F87F512/data/Containers/Bundle/Application/3F282177-5E2C-4838-987B-507586C1BC52/MyInterpretedApp.app/libmonosgen-2.0.dylib : sigabrt_signal_handler
	0x7fff6fafbdfd - /usr/lib/system/libsystem_platform.dylib : _sigtramp
	0x600003965800 - Unknown
	0x7fff2010b6b7 - /Applications/Xcode_13.3.0.app/Contents/Developer/Platforms/iPhoneOS.platform/Library/Developer/CoreSimulator/Profiles/Runtimes/iOS.simruntime/Contents/Resources/RuntimeRoot/usr/lib/system/libsystem_c.dylib : abort
	0x10b0b69c9 - /Users/rolf/Library/Developer/CoreSimulator/Devices/D21038CB-BC03-4F58-886D-D8106F87F512/data/Containers/Bundle/Application/3F282177-5E2C-4838-987B-507586C1BC52/MyInterpretedApp.app/libmonosgen-2.0.dylib : mono_log_write_os_log
	0x10b27d628 - /Users/rolf/Library/Developer/CoreSimulator/Devices/D21038CB-BC03-4F58-886D-D8106F87F512/data/Containers/Bundle/Application/3F282177-5E2C-4838-987B-507586C1BC52/MyInterpretedApp.app/libmonosgen-2.0.dylib : monoeg_g_logv
	0x10b27d7c2 - /Users/rolf/Library/Developer/CoreSimulator/Devices/D21038CB-BC03-4F58-886D-D8106F87F512/data/Containers/Bundle/Application/3F282177-5E2C-4838-987B-507586C1BC52/MyInterpretedApp.app/libmonosgen-2.0.dylib : monoeg_g_log
	0x10b1b2ea9 - /Users/rolf/Library/Developer/CoreSimulator/Devices/D21038CB-BC03-4F58-886D-D8106F87F512/data/Containers/Bundle/Application/3F282177-5E2C-4838-987B-507586C1BC52/MyInterpretedApp.app/libmonosgen-2.0.dylib : load_aot_module
	0x10afe14cd - /Users/rolf/Library/Developer/CoreSimulator/Devices/D21038CB-BC03-4F58-886D-D8106F87F512/data/Containers/Bundle/Application/3F282177-5E2C-4838-987B-507586C1BC52/MyInterpretedApp.app/libmonosgen-2.0.dylib : mono_assembly_request_load_from
	0x10afe0ecd - /Users/rolf/Library/Developer/CoreSimulator/Devices/D21038CB-BC03-4F58-886D-D8106F87F512/data/Containers/Bundle/Application/3F282177-5E2C-4838-987B-507586C1BC52/MyInterpretedApp.app/libmonosgen-2.0.dylib : mono_assembly_request_open
	0x10afe2ed0 - /Users/rolf/Library/Developer/CoreSimulator/Devices/D21038CB-BC03-4F58-886D-D8106F87F512/data/Containers/Bundle/Application/3F282177-5E2C-4838-987B-507586C1BC52/MyInterpretedApp.app/libmonosgen-2.0.dylib : mono_assembly_open
	0x10aa675f1 - /Users/rolf/Library/Developer/CoreSimulator/Devices/D21038CB-BC03-4F58-886D-D8106F87F512/data/Containers/Bundle/Application/3F282177-5E2C-4838-987B-507586C1BC52/MyInterpretedApp.app/libxamarin-dotnet-debug.dylib : xamarin_assembly_preload_hook
	0x10afe36a7 - /Users/rolf/Library/Developer/CoreSimulator/Devices/D21038CB-BC03-4F58-886D-D8106F87F512/data/Containers/Bundle/Application/3F282177-5E2C-4838-987B-507586C1BC52/MyInterpretedApp.app/libmonosgen-2.0.dylib : invoke_assembly_preload_hook
	0x10afe3889 - /Users/rolf/Library/Developer/CoreSimulator/Devices/D21038CB-BC03-4F58-886D-D8106F87F512/data/Containers/Bundle/Application/3F282177-5E2C-4838-987B-507586C1BC52/MyInterpretedApp.app/libmonosgen-2.0.dylib : mono_assembly_load_corlib

vargaz · 2022-05-12T06:56:17Z

Not sure what is happening here, MONO_AOT_MODE_INTERP_ONLY is the same as running with --interp.

vargaz · 2022-05-12T07:04:07Z

Maybe there is a mismatch in the headers, this is the definition of MonoAotMode:
https://github.com/dotnet/runtime/blob/main/src/native/public/mono/jit/details/jit-types.h#L15
MONO_AOT_MODE_INTERP_ONLY has value 8.

rolfbjarne · 2022-05-12T16:19:49Z

Maybe there is a mismatch in the headers,

That was the issue, once I fixed the value, using MONO_AOT_MODE_INTERP_ONLY works.

Now we just need to decide whether that's what we want to do :)

vargaz · 2022-05-20T17:17:26Z

Implementing infinite trampolines is a lot of work, so I'd prefer this approach.

steveisok · 2022-07-05T14:45:01Z

Closing as I believe MONO_AOT_MODE_INTERP_ONLY is the way to go. Please reopen if anyone disagrees.

… for x86_64. The 'MONO_AOT_MODE_INTERP' mode is based on full AOT, which requires infinite trampoline code to not potentially run out of trampolines. However, that mode isn't implemented for x86_64, so in that case use 'MONO_AOT_MODE_INTERP_ONLY', which will use the JIT as a fallback. Ref: dotnet/runtime#68808 Fixes: xamarin#14887

… for x86_64. (#15429) The 'MONO_AOT_MODE_INTERP' mode is based on full AOT, which requires infinite trampoline code to not potentially run out of trampolines. However, that mode isn't implemented for x86_64, so in that case use 'MONO_AOT_MODE_INTERP_ONLY', which will use the JIT as a fallback. Ref: dotnet/runtime#68808 Fixes: #14887

SimenCTan · 2022-07-20T15:15:12Z

I have the same issue when run maui on ios.
according to this Pull request
how to add --aot arguments in MtouchExtraArgs when building a .NET project @rolfbjarne

ERR Info：

Ran out of trampolines of type 0 in
 '*/CoreSimulator/Devices/F06D2CC0-2EEA-4C06-B672-721C0A277327/
data/Containers/Bundle/Application/C9EA73F5-9801-4F0C-93D3-302108B93249/
*.app/System.Private.CoreLib.dll'  (limit 4096). 
See http://docs.xamarin.com/ios/troubleshooting for instructions on how to fix this condition.

SimenCTan · 2022-08-11T17:07:51Z

After updating dotnet sdk 6.0.400 this issue still exist，is there an update ?
@vargaz @rolfbjarne

rolfbjarne · 2022-08-17T15:24:08Z

@SimenCTan The fix didn't make it to 6.0.400, and right now it's slated for the .NET 7 release later this fall.

Unfortunately the only workaround at the moment is to disable the interpreter.

SimenCTan · 2022-08-18T02:13:24Z

@SimenCTan The fix didn't make it to 6.0.400, and right now it's slated for the .NET 7 release later this fall.

Unfortunately the only workaround at the moment is to disable the interpreter.

thx, The temporary solution is add
<UseInterpreter Condition="'$(Configuration)' == 'Debug' AND '$(TargetFramework)' != 'net6.0-ios'">True</UseInterpreter> to the .csproj file

ghost added the untriaged New issue has not been triaged by the area owner label May 3, 2022

dotnet-issue-labeler bot added the area-Codegen-AOT-mono label May 3, 2022

rolfbjarne mentioned this issue May 3, 2022

[net6][iOS] Ran out of trampolines of type 1 running on x64 simulator xamarin/xamarin-macios#14855

Closed

marek-safar assigned vargaz May 3, 2022

rolfbjarne mentioned this issue May 4, 2022

Insufficient number of trampolines with the interpreter enabled on x86_64 xamarin/xamarin-macios#14887

Closed

rolfbjarne mentioned this issue May 4, 2022

.NET 6: documentation xamarin/xamarin-macios#10574

Closed

jeromelaban mentioned this issue Jun 23, 2022

[MacOS] The AOT compiler '' does not exist. unoplatform/uno#9093

Closed

steveisok removed the untriaged New issue has not been triaged by the area owner label Jul 5, 2022

steveisok closed this as completed Jul 5, 2022

rolfbjarne mentioned this issue Jul 8, 2022

[tools] Use MONO_AOT_MODE_INTERP_ONLY instead of MONO_AOT_MODE_INTERP for x86_64. xamarin/xamarin-macios#15429

Merged

SimenCTan mentioned this issue Jul 25, 2022

Ran out of trampolines of type 0 running on iPhone iOS 15.5 simulator dotnet/maui#8970

Closed

ghost locked as resolved and limited conversation to collaborators Sep 17, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[net6][iOS] `Ran out of trampolines of type 1` running on x64 simulator #68808

[net6][iOS] `Ran out of trampolines of type 1` running on x64 simulator #68808

rolfbjarne commented May 3, 2022

rolfbjarne commented May 3, 2022

rolfbjarne commented May 3, 2022

rolfbjarne commented May 3, 2022

rolfbjarne commented May 3, 2022

rolfbjarne commented May 3, 2022

rolfbjarne commented May 3, 2022

vargaz commented May 3, 2022

vargaz commented May 3, 2022

steveisok commented May 3, 2022

rolfbjarne commented May 4, 2022

rolfbjarne commented May 4, 2022

vargaz commented May 5, 2022

vargaz commented May 5, 2022

steveisok commented May 5, 2022

vargaz commented May 5, 2022

lambdageek commented May 5, 2022

rolfbjarne commented May 11, 2022

steveisok commented May 11, 2022

rolfbjarne commented May 11, 2022

vargaz commented May 12, 2022

vargaz commented May 12, 2022

rolfbjarne commented May 12, 2022

vargaz commented May 20, 2022

steveisok commented Jul 5, 2022

SimenCTan commented Jul 20, 2022 •

edited

Loading

SimenCTan commented Aug 11, 2022

rolfbjarne commented Aug 17, 2022

SimenCTan commented Aug 18, 2022 •

edited

Loading

[net6][iOS] Ran out of trampolines of type 1 running on x64 simulator #68808

[net6][iOS] Ran out of trampolines of type 1 running on x64 simulator #68808

Comments

rolfbjarne commented May 3, 2022

Steps to Reproduce

Expected Behavior

Actual Behavior

Environment

Build Logs

Other details

rolfbjarne commented May 3, 2022

rolfbjarne commented May 3, 2022

rolfbjarne commented May 3, 2022

rolfbjarne commented May 3, 2022

rolfbjarne commented May 3, 2022

rolfbjarne commented May 3, 2022

vargaz commented May 3, 2022

vargaz commented May 3, 2022

steveisok commented May 3, 2022

rolfbjarne commented May 4, 2022

rolfbjarne commented May 4, 2022

vargaz commented May 5, 2022

vargaz commented May 5, 2022

steveisok commented May 5, 2022

vargaz commented May 5, 2022

lambdageek commented May 5, 2022

rolfbjarne commented May 11, 2022

steveisok commented May 11, 2022

rolfbjarne commented May 11, 2022

vargaz commented May 12, 2022

vargaz commented May 12, 2022

rolfbjarne commented May 12, 2022

vargaz commented May 20, 2022

steveisok commented Jul 5, 2022

SimenCTan commented Jul 20, 2022 • edited Loading

SimenCTan commented Aug 11, 2022

rolfbjarne commented Aug 17, 2022

SimenCTan commented Aug 18, 2022 • edited Loading

[net6][iOS] `Ran out of trampolines of type 1` running on x64 simulator #68808

[net6][iOS] `Ran out of trampolines of type 1` running on x64 simulator #68808

SimenCTan commented Jul 20, 2022 •

edited

Loading

SimenCTan commented Aug 18, 2022 •

edited

Loading