[arm64] Crash in Roslyn

[lewurm]

Steps to Reproduce

1. checkout mono on e.g. Cavium ThunderX SoC
2. run make and get a roslyn crash. it's very reliable

same workaround as mentioned in a potentially related bug applies: https://bugzilla.xamarin.com/show_bug.cgi?id=56546#c2

Current Behavior

Thread 1 throws an unhandled exception:

(lldb) bt
* thread #1, name = 'mono', stop reason = signal SIGSTOP
  * frame #0: 0x00000000005014e4 mono-sgen`mono_handle_exception_internal(ctx=0x0000ffffda076af0, obj=0x0000ffff92d2fc80, resume=0, out_ji=0x0000000000000000) at mini-exceptions.c:2013
    frame #1: 0x0000000000502128 mono-sgen`mono_handle_exception(ctx=0x0000ffffda076af0, obj=0x0000ffff92d2fc80) at mini-exceptions.c:2363
    frame #2: 0x000000000057405c mono-sgen`mono_arm_throw_exception(arg=0x0000ffff92d2fc80, pc=281473102538176, int_regs=0x0000ffffda076e20, fp_regs=0x0000ffffda076f20, corlib=0, rethrow=1) at exceptions-arm64.c:410
    frame #3: 0x0000ffff932b1ea4
    frame #4: 0x0000ffff904a65c4
    frame #5: 0x0000ffff8a759510
    frame #6: 0x0000ffff88f286b8
    frame #7: 0x0000ffff88f27c7c
    frame #8: 0x0000ffff8a717058
    frame #9: 0x0000ffff8a714cd0
    frame #10: 0x0000ffff89f18fc4
    frame #11: 0x0000ffff8a75b1c4
    frame #12: 0x0000ffff9049f76c
    frame #13: 0x0000ffff904b7518
    frame #14: 0x0000ffff904b743c
    frame #15: 0x0000ffff904b732c
    frame #16: 0x0000ffff9302ef94
    frame #17: 0x0000ffff9302f164
    frame #18: 0x0000ffff9302ed64
    frame #19: 0x0000ffff93041794
    frame #20: 0x0000ffff931bb3b4
    frame #21: 0x0000ffff931b494c
    frame #22: 0x0000ffff931b4680
    frame #23: 0x0000ffff931b4810
    frame #24: 0x0000000000421a74 mono-sgen`mono_jit_runtime_invoke(method=0x000000002d7a6548, obj=0x0000000000000000, params=0x0000ffffda078270, exc=0x0000000000000000, error=0x0000ffffda078360) at mini-runtime.c:2807
    frame #25: 0x00000000006b8488 mono-sgen`do_runtime_invoke(method=0x000000002d7a6548, obj=0x0000000000000000, params=0x0000ffffda078270, exc=0x0000000000000000, error=0x0000ffffda078360) at object.c:2923
    frame #26: 0x00000000006b86e8 mono-sgen`mono_runtime_invoke_checked(method=0x000000002d7a6548, obj=0x0000000000000000, params=0x0000ffffda078270, error=0x0000ffffda078360) at object.c:3076
    frame #27: 0x00000000006bc418 mono-sgen`do_exec_main_checked(method=0x000000002d7a6548, args=0x0000ffff92c003e8, error=0x0000ffffda078360) at object.c:4820
    frame #28: 0x00000000006bc7d0 mono-sgen`mono_runtime_exec_main_checked(method=0x000000002d7a6548, args=0x0000ffff92c003e8, error=0x0000ffffda078360) at object.c:4921
    frame #29: 0x00000000006bb0ac mono-sgen`mono_runtime_run_main_checked(method=0x000000002d7a6548, argc=47, argv=0x0000ffffda078770, error=0x0000ffffda078360) at object.c:4342
    frame #30: 0x00000000004bbf28 mono-sgen`mono_jit_exec(domain=0x000000002d7376d0, assembly=0x000000002d7a6d90, argc=47, argv=0x0000ffffda078770) at driver.c:1202
    frame #31: 0x00000000004bc2b4 mono-sgen`main_thread_handler(user_data=0x0000ffffda078598) at driver.c:1279
    frame #32: 0x00000000004bf550 mono-sgen`mono_main(argc=50, argv=0x0000ffffda078758) at driver.c:2380
    frame #33: 0x0000000000419a7c mono-sgen`mono_main_with_options(argc=50, argv=0x0000ffffda078758) at main.c:50
    frame #34: 0x000000000041a6cc mono-sgen`main(argc=50, argv=0x0000ffffda078758) at main.c:398
    frame #35: 0x0000ffff934b58a0 libc.so.6`__libc_start_main(main=0x0000000000000000, argc=0, argv=0x0000000000000000, init=<unavailable>, fini=<unavailable>, rtld_fini=<unavailable>, stack_end=<unavailable>) at libc-start.c:291
    frame #36: 0x0000000000419708 mono-sgen`_start + 40

managed stack trace (cut off):

[ERROR] FATAL UNHANDLED EXCEPTION: System.AggregateException: One or more errors occurred. ---> System.AggregateException: One or more errors occurred. ---> System.AggregateException: One or more errors occurred. ---> System.AggregateException: One or more errors occurred. ---> System.IndexOutOfRangeException: Index was outside the bounds of the array.
  at System.Collections.Immutable.ImmutableArray`1+Builder[T].get_Item (System.Int32 index) [0x00009] in <36486b016d234fca8cd67892bf29c7b5>:0
  at Microsoft.CodeAnalysis.PooledObjects.ArrayBuilder`1[T].get_Item (System.Int32 index) [0x00000] in <cdd7d460203b4181bde2f531be3af532>:0
  at Microsoft.CodeAnalysis.CSharp.Binder.ResultSymbol (Microsoft.CodeAnalysis.CSharp.LookupResult result, System.String simpleName, System.Int32 arity, Microsoft.CodeAnalysis.SyntaxNode where, Microsoft.CodeAnalysis.DiagnosticBag diagnostics, System.Boolean suppressUseSiteDiagnostics, System.Boolean& wasError, Microsoft.CodeAnalysis.CSharp.Symbols.NamespaceOrTypeSymbol qualifierOpt, Microsoft.CodeAnalysis.CSharp.LookupOptions options) [0x006a5] in <c5f8aa12db2342148d6752195df5f666>:0
  at Microsoft.CodeAnalysis.CSharp.Binder.BindNonGenericSimpleNamespaceOrTypeOrAliasSymbol (Microsoft.CodeAnalysis.CSharp.Syntax.IdentifierNameSyntax node, Microsoft.CodeAnalysis.DiagnosticBag diagnostics, Roslyn.Utilities.ConsList`1[T] basesBeingResolved, System.Boolean suppressUseSiteDiagnostics, Microsoft.CodeAnalysis.CSharp.Symbols.NamespaceOrTypeSymbol qualifierOpt, System.Boolean isNameofArgument, Microsoft.CodeAnalysis.PooledObjects.ArrayBuilder`1[T] symbols) [0x00146] in <c5f8aa12db2342148d6752195df5f666>:0
  at Microsoft.CodeAnalysis.CSharp.Binder.BindNamespaceOrTypeOrAliasSymbol (Microsoft.CodeAnalysis.CSharp.Syntax.ExpressionSyntax syntax, Microsoft.CodeAnalysis.DiagnosticBag diagnostics, Roslyn.Utilities.ConsList`1[T] basesBeingResolved, System.Boolean suppressUseSiteDiagnostics) [0x000d4] in <c5f8aa12db2342148d6752195df5f666>:0
  at Microsoft.CodeAnalysis.CSharp.Binder.BindTypeOrAlias (Microsoft.CodeAnalysis.CSharp.Syntax.ExpressionSyntax syntax, Microsoft.CodeAnalysis.DiagnosticBag diagnostics, Roslyn.Utilities.ConsList`1[T] basesBeingResolved) [0x00000] in <c5f8aa12db2342148d6752195df5f666>:0
  at Microsoft.CodeAnalysis.CSharp.Binder.BindType (Microsoft.CodeAnalysis.CSharp.Syntax.ExpressionSyntax syntax, Microsoft.CodeAnalysis.DiagnosticBag diagnostics, Roslyn.Utilities.ConsList`1[T] basesBeingResolved) [0x00000] in <c5f8aa12db2342148d6752195df5f666>:0
  at Microsoft.CodeAnalysis.CSharp.Symbols.ParameterHelpers.MakeParameters (Microsoft.CodeAnalysis.CSharp.Binder binder, Microsoft.CodeAnalysis.CSharp.Symbol owner, Microsoft.CodeAnalysis.CSharp.Syntax.BaseParameterListSyntax syntax, Microsoft.CodeAnalysis.SyntaxToken& arglistToken, Microsoft.CodeAnalysis.DiagnosticBag diagnostics, System.Boolean allowRefOrOut, System.Boolean allowThis, System.Boolean addRefReadOnlyModifier) [0x000fc] in <c5f8aa12db2342148d6752195df5f666>:0
  at Microsoft.CodeAnalysis.CSharp.Symbols.SourceConstructorSymbol.MethodChecks (Microsoft.CodeAnalysis.DiagnosticBag diagnostics) [0x0002e] in <c5f8aa12db2342148d6752195df5f666>:0
[...]

Expected Behavior

do not crash.

On which platforms did you notice this

[ ] macOS
[x] Linux, arm64
[ ] Windows

Version Used:

recent master, 06b836e

Ask @akoeplinger in order to get access to that machine.

[BrzVlad]

Investigated this issue, submitted issue to roslyn team.

dotnet/roslyn#24932

[akoeplinger]

Looks like this also happens with Realtek RTD1295 Quad-Core ARM Cortex-A53 Processor @ 1.4GHz from #7382 (comment)

[jaykrell]

The memory model still.
"Fix" the memory model or fix Roslyn, and any other similar code, if there is any.
#8731

[IanusInferus]

For those who encountered this problem on MSBuild, I found a workaround.
Roslyn targets file can be patched for /parallel- by replacing

Sources="@(Compile)"

with

Sources="@(Compile);/parallel-"

in /usr/local/lib/mono/msbuild/Current/bin/Roslyn/Microsoft.CSharp.Core.targets and /usr/local/lib/mono/msbuild/Current/bin/Roslyn/Microsoft.VisualBasic.Core.targets. (Assuming mono is installed in /usr/local.)

[dch]

also reported on Ampere eMAG (32 core armv8), work-around above is perfect. Not clear what the impact of appending ;/parallel- is though?

[IanusInferus]

also reported on Ampere eMAG (32 core armv8), work-around above is perfect. Not clear what the impact of appending ;/parallel- is though?

It means that Roslyn will run single-threaded and will not trigger any problems related to the memory ordering model dismatch between x86(Total store order) and ARM(Weak memory order). So the compiling will be slower, but it will not crash.

[dch]

On Tue, 1 Feb 2022, at 10:22, Ianus Inferus wrote: > also reported on Ampere eMAG (32 core armv8), work-around above is perfect. Not clear what the impact of appending `;/parallel-` is though? > It means that Roslyn will run single-threaded and will not trigger any problems related to the memory ordering model dismatch between x86(Total store order) and ARM(Weak memory order). So the compiling will be slower, but it will not crash.

thank-you! I assume this is only a compile-time issue, then, and not latet at runtime.

[IanusInferus]

On Tue, 1 Feb 2022, at 10:22, Ianus Inferus wrote: > also reported on Ampere eMAG (32 core armv8), work-around above is perfect. Not clear what the impact of appending ;/parallel- is though? > It means that Roslyn will run single-threaded and will not trigger any problems related to the memory ordering model dismatch between x86(Total store order) and ARM(Weak memory order). So the compiling will be slower, but it will not crash.
thank-you! I assume this is only a compile-time issue, then, and not latet at runtime.

Yes.
As long as your application do locking and atomics assuming weak memory order, or it doesn't use multi-threading, this is not a problem at runtime.
If an application falls into the same trap as Roslyn, the fix has to be done in the source code, or on the hardware (like Apple M1's runtime toggle for TSO). Mono can not "fix" this problem with reasonable performance penalty.

[BrzVlad]

This shouldn't really be an issue since #17136. I'm assuming you are using an older version of mono.