Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

condition 'ji' not met, with 'dynamic' and multithreading #14080

Closed
kdg3737 opened this issue Apr 16, 2019 · 19 comments · Fixed by #16589
Closed

condition 'ji' not met, with 'dynamic' and multithreading #14080

kdg3737 opened this issue Apr 16, 2019 · 19 comments · Fixed by #16589

Comments

@kdg3737
Copy link

@kdg3737 kdg3737 commented Apr 16, 2019

Steps to Reproduce

  1. Create a method with a 'dynamic' type, my method is an indexer of a custom ExpandoObject and looks like 'public dynamic this[string key] { get { ... } set { ... }'
  2. Call it in a loop from multiple threads, make sure to start the threads at the same time (the problem disappears with start-delay between the threads)
  3. Watch the program crash 10-20% of the time with the exception/stack-trace below

Current Behavior

Crashes, SIGABRT. I believe it is the same issue as: http://www.johankarlsson.net/2017/10/android-8-and-condition-ji-not-met.html

Expected Behavior

Doesn't crash on Windows/.Net Framework.

On which platforms did you notice this

[ ] macOS
[X ] Linux
[ ] Windows

Version Used:

6.3, master, arm64

Stacktrace

* Assertion at mini-arm64.c:886, condition `ji' not met


=================================================================
	Native Crash Reporting
=================================================================
Got a SIGABRT while executing native code. This usually indicates
a fatal error in the mono runtime or one of the native libraries 
used by your application.
=================================================================
/proc/self/maps:
5588609000-55889be000 r-xp 00000000 b3:07 76365                          /usr/local/bin/mono-sgen
55889cd000-55889d5000 r--p 003b4000 b3:07 76365                          /usr/local/bin/mono-sgen
55889d5000-55889db000 rw-p 003bc000 b3:07 76365                          /usr/local/bin/mono-sgen
55889db000-55889f2000 rw-p 00000000 00:00 0 
558de35000-558eba5000 rw-p 00000000 00:00 0                              [heap]
7ed8fd0000-7ed9e68000 rw-p 00000000 00:00 0 
7ed9e68000-7eda16e000 r--p 00000000 b3:07 166160                         /usr/local/lib/mono/gac/System.Web/4.0.0.0__b03f5f7f11d50a3a/System.Web.dll
7eda16e000-7eda16f000 ---p 00000000 00:00 0 
7eda16f000-7eda36f000 rw-p 00000000 00:00 0 
7eda36f000-7eda370000 ---p 00000000 00:00 0 
7eda370000-7eda570000 rw-p 00000000 00:00 0 
7eda570000-7eda571000 ---p 00000000 00:00 0 
7eda571000-7eda771000 rw-p 00000000 00:00 0 
7eda771000-7eda772000 ---p 00000000 00:00 0 
7eda772000-7eda972000 rw-p 00000000 00:00 0 
7eda972000-7eda973000 ---p 00000000 00:00 0 
7eda973000-7edab73000 rw-p 00000000 00:00 0 
7edab73000-7edab74000 ---p 00000000 00:00 0 
7edab74000-7edad74000 rw-p 00000000 00:00 0 
7edad74000-7edad75000 ---p 00000000 00:00 0 
7edad75000-7edaf75000 rw-p 00000000 00:00 0 
7edaf75000-7edaf76000 ---p 00000000 00:00 0 
7edaf76000-7edbc00000 rw-p 00000000 00:00 0 
7edbc75000-7edbc76000 ---p 00000000 00:00 0 
7edbc76000-7edc900000 rw-p 00000000 00:00 0 

=================================================================
	Native stacktrace:
=================================================================
	0x55886ab28c - mono : (null)

=================================================================
	Telemetry Dumper:
=================================================================
Pkilling 0x7f617ff1d0 from 0x7eda56f1d0
Pkilling 0x7f244ff1d0 from 0x7eda56f1d0
Pkilling 0x7f6340b1d0 from 0x7eda56f1d0
Pkilling 0x7eeffff1d0 from 0x7eda56f1d0
Pkilling 0x7eda9711d0 from 0x7eda56f1d0
Pkilling 0x7f601231d0 from 0x7eda56f1d0
Pkilling 0x7f612ff1d0 from 0x7eda56f1d0
Pkilling 0x7edab721d0 from 0x7eda56f1d0
Pkilling 0x7f804b9a40 from 0x7eda56f1d0
Pkilling 0x7f60eff1d0 from 0x7eda56f1d0
Pkilling 0x7edbe751d0 from 0x7eda56f1d0
Pkilling 0x7eecd751d0 from 0x7eda56f1d0
Pkilling 0x7f5eabc1d0 from 0x7eda56f1d0
Pkilling 0x7edf2751d0 from 0x7eda56f1d0
Pkilling 0x7edad731d0 from 0x7eda56f1d0
Pkilling 0x7f63e731d0 from 0x7eda56f1d0
Pkilling 0x7f5d66c1d0 from 0x7eda56f1d0
Pkilling 0x7edaf741d0 from 0x7eda56f1d0
Pkilling 0x7f605ff1d0 from 0x7eda56f1d0
Pkilling 0x7f0a3611d0 from 0x7eda56f1d0
Pkilling 0x7eef4741d0 from 0x7eda56f1d0
Pkilling 0x7f71b1f1d0 from 0x7eda56f1d0
Pkilling 0x7f71ebb1d0 from 0x7eda56f1d0
Pkilling 0x7edb1751d0 from 0x7eda56f1d0
Pkilling 0x7ede5751d0 from 0x7eda56f1d0
Pkilling 0x7f5c6ff1d0 from 0x7eda56f1d0
Pkilling 0x7f720bc1d0 from 0x7eda56f1d0
Pkilling 0x7f5c2ff1d0 from 0x7eda56f1d0
Pkilling 0x7f722bd1d0 from 0x7eda56f1d0
Pkilling 0x7f60a571d0 from 0x7eda56f1d0
Pkilling 0x7f63b011d0 from 0x7eda56f1d0
Pkilling 0x7f097d61d0 from 0x7eda56f1d0
Pkilling 0x7edd8751d0 from 0x7eda56f1d0
Pkilling 0x7eee7751d0 from 0x7eda56f1d0
Pkilling 0x7f7034d1d0 from 0x7eda56f1d0
Pkilling 0x7edffff1d0 from 0x7eda56f1d0
Pkilling 0x7eda36e1d0 from 0x7eda56f1d0
Pkilling 0x7f7054e1d0 from 0x7eda56f1d0
Pkilling 0x7f5e6ff1d0 from 0x7eda56f1d0
Pkilling 0x7f7074f1d0 from 0x7eda56f1d0
Pkilling 0x7f242fe1d0 from 0x7eda56f1d0
Pkilling 0x7f08c4b1d0 from 0x7eda56f1d0
Pkilling 0x7eeda751d0 from 0x7eda56f1d0
Pkilling 0x7f7d6ca1d0 from 0x7eda56f1d0
Pkilling 0x7edcb751d0 from 0x7eda56f1d0
Pkilling 0x7eda7701d0 from 0x7eda56f1d0
Pkilling 0x7f3c2ff1d0 from 0x7eda56f1d0

=================================================================
	External Debugger Dump:
=================================================================
mono_gdb_render_native_backtraces not supported on this platform, unable to find gdb or lldb
		PAGE REQUESTED: /Default.aspx

=================================================================
	Basic Fault Adddress Reporting
=================================================================
Memory around native instruction pointer (0x7f8023d9fc):0x7f8023d9ec  02 00 80 d2 03 01 80 d2 e8 10 80 d2 01 00 00 d4  ................
0x7f8023d9fc  f3 0b 40 f9 e0 03 04 2a fd 7b d2 a8 c0 03 5f d6  ..@....*.{...._.
0x7f8023da0c  1f 20 03 d5 81 08 00 f0 21 20 47 f9 42 d0 3b d5  . ......! G.B.;.
0x7f8023da1c  e0 03 00 4b 04 00 80 12 40 68 21 b8 ef ff ff 17  ...K....@h!.....

=================================================================
	Managed Stacktrace:
=================================================================
=================================================================
Aborted
@lambdageek

This comment has been minimized.

Copy link
Member

@lambdageek lambdageek commented Apr 16, 2019

@kdg3737 Could you share a complete reproduction example.

Also if possible, could you install gdb or lldb on your Linux system and reproduce the crash again - if there's a debugger installed, Mono will collect a native stacktrace that may be helpful in further diagnosing this issue.

@kdg3737

This comment has been minimized.

Copy link
Author

@kdg3737 kdg3737 commented Apr 17, 2019

The following code in a console application should do it:

        [STAThread]
        static void Main(string[] args)
        {
            System.Threading.ThreadStart ts = () =>
            {
                double sum = 0;
                for (int i = 0; i < 10000; i++)
                {
                    dynamic expo = new System.Dynamic.ExpandoObject();
                    expo.A = sum;
                    sum += expo.A + 37;
                }
            };
            for (int i = 0; i < 50; i++)
            {
                System.Threading.Thread t = new System.Threading.Thread(ts);
                t.Start();
            }
            System.Console.WriteLine("Waiting for it...");

            return;
        }

I'm using a bash script to execute it in a loop, because it doesn't happen all the time:

#!/bin/bash

for i in `seq 1 10000`;
     MONO_DEBUG=no-compact-seq-points mono --debug YOUREXE.exe
done  

I installed gdb, the output now looks like:

> * Assertion at mini-arm64.c:886, condition `ji' not met
> 
> 
> =================================================================
> 	Native Crash Reporting
> =================================================================
> Got a SIGABRT while executing native code. This usually indicates
> a fatal error in the mono runtime or one of the native libraries 
> used by your application.
> =================================================================
> /proc/self/maps:
> 5593996000-5593d4b000 r-xp 00000000 b3:07 76365                          /usr/local/bin/mono-sgen
> 5593d5a000-5593d62000 r--p 003b4000 b3:07 76365                          /usr/local/bin/mono-sgen
> 5593d62000-5593d68000 rw-p 003bc000 b3:07 76365                          /usr/local/bin/mono-sgen
> 5593d68000-5593d7f000 rw-p 00000000 00:00 0 
> 559dde7000-559df89000 rw-p 00000000 00:00 0                              [heap]
> 7f01e00000-7f01f00000 rw-p 00000000 00:00 0 
> 7f01ff0000-7f01ff1000 ---p 00000000 00:00 0 
> 7f01ff1000-7f021f1000 rw-p 00000000 00:00 0 
> 7f021f1000-7f021f2000 ---p 00000000 00:00 0 
> 7f021f2000-7f023f2000 rw-p 00000000 00:00 0 
> 7f023f2000-7f023f3000 ---p 00000000 00:00 0 
> 7f023f3000-7f025f3000 rw-p 00000000 00:00 0 
> 7f025f3000-7f025f4000 ---p 00000000 00:00 0 
> 7f025f4000-7f027f4000 rw-p 00000000 00:00 0 
> 7f027f4000-7f027f5000 ---p 00000000 00:00 0 
> 7f027f5000-7f029f5000 rw-p 00000000 00:00 0 
> 7f029f5000-7f029f6000 ---p 00000000 00:00 0 
> 7f029f6000-7f02bf6000 rw-p 00000000 00:00 0 
> 7f02bf6000-7f02bf7000 ---p 00000000 00:00 0 
> 7f02bf7000-7f02df7000 rw-p 00000000 00:00 0 
> 7f02df7000-7f02df8000 ---p 00000000 00:00 0 
> 7f02df8000-7f02ff8000 rw-p 00000000 00:00 0 
> 7f02ff8000-7f02ff9000 ---p 00000000 00:00 0 
> 7f02ff9000-7f031f9000 rw-p 00000000 00:00 0 
> 7f031f9000-7f031fa000 ---p 00000000 00:00 0 
> 
> =================================================================
> 	Native stacktrace:
> =================================================================
> 	0x5593a3828c - mono : (null)
> 
> =================================================================
> 	Telemetry Dumper:
> =================================================================
> Pkilling 0x7f761f01d0 from 0x7f74fe71d0
> Pkilling 0x7f777fb1d0 from 0x7f74fe71d0
> Pkilling 0x7f88bbca40 from 0x7f74fe71d0
> Pkilling 0x7f74de61d0 from 0x7f74fe71d0
> Pkilling 0x7f035fa1d0 from 0x7f74fe71d0
> Pkilling 0x7f763f11d0 from 0x7f74fe71d0
> Pkilling 0x7f779fc1d0 from 0x7f74fe71d0
> Pkilling 0x7f021f01d0 from 0x7f74fe71d0
> Pkilling 0x7f037fb1d0 from 0x7f74fe71d0
> Pkilling 0x7f765f21d0 from 0x7f74fe71d0
> Pkilling 0x7f77bfd1d0 from 0x7f74fe71d0
> Pkilling 0x7f023f11d0 from 0x7f74fe71d0
> Pkilling 0x7f751e81d0 from 0x7f74fe71d0
> Pkilling 0x7f039fc1d0 from 0x7f74fe71d0
> Pkilling 0x7f767f31d0 from 0x7f74fe71d0
> Pkilling 0x7f77dfe1d0 from 0x7f74fe71d0
> Pkilling 0x7f025f21d0 from 0x7f74fe71d0
> Pkilling 0x7f753e91d0 from 0x7f74fe71d0
> Pkilling 0x7f03bfd1d0 from 0x7f74fe71d0
> Pkilling 0x7f769f41d0 from 0x7f74fe71d0
> Pkilling 0x7f77fff1d0 from 0x7f74fe71d0
> Pkilling 0x7f027f31d0 from 0x7f74fe71d0
> Pkilling 0x7f755ea1d0 from 0x7f74fe71d0
> Pkilling 0x7f03dfe1d0 from 0x7f74fe71d0
> Pkilling 0x7f76bf51d0 from 0x7f74fe71d0
> Pkilling 0x7f843de1d0 from 0x7f74fe71d0
> Pkilling 0x7f029f41d0 from 0x7f74fe71d0
> Pkilling 0x7f757eb1d0 from 0x7f74fe71d0
> Pkilling 0x7f03fff1d0 from 0x7f74fe71d0
> Pkilling 0x7f76df61d0 from 0x7f74fe71d0
> Pkilling 0x7f85cc31d0 from 0x7f74fe71d0
> Pkilling 0x7f845df1d0 from 0x7f74fe71d0
> Pkilling 0x7f743e11d0 from 0x7f74fe71d0
> Pkilling 0x7f02bf51d0 from 0x7f74fe71d0
> Pkilling 0x7f759ec1d0 from 0x7f74fe71d0
> Pkilling 0x7f76ff71d0 from 0x7f74fe71d0
> Pkilling 0x7f847e01d0 from 0x7f74fe71d0
> Pkilling 0x7f745e21d0 from 0x7f74fe71d0
> Pkilling 0x7f02df61d0 from 0x7f74fe71d0
> Pkilling 0x7f75bed1d0 from 0x7f74fe71d0
> Pkilling 0x7f771f81d0 from 0x7f74fe71d0
> Pkilling 0x7f747e31d0 from 0x7f74fe71d0
> Pkilling 0x7f02ff71d0 from 0x7f74fe71d0
> Pkilling 0x7f75dee1d0 from 0x7f74fe71d0
> Pkilling 0x7f773f91d0 from 0x7f74fe71d0
> Pkilling 0x7f749e41d0 from 0x7f74fe71d0
> Pkilling 0x7f031f81d0 from 0x7f74fe71d0
> Pkilling 0x7f75fef1d0 from 0x7f74fe71d0
> Pkilling 0x7f775fa1d0 from 0x7f74fe71d0
> Pkilling 0x7f74be51d0 from 0x7f74fe71d0
> Pkilling 0x7f033f91d0 from 0x7f74fe71d0
> Entering thread summarizer pause from 0x7f74fe71d0
> Finished thread summarizer pause from 0x7f74fe71d0.
> Crash Reporter has timed out, sending SIGSEGV

Mono also creates .blob and .json files, you need those?

@kdg3737

This comment has been minimized.

Copy link
Author

@kdg3737 kdg3737 commented Apr 17, 2019

Another error that's showing up running the example (not sure if it's related):

Unhandled Exception:
System.ArgumentNullException: Value cannot be null.
Parameter name: returnLabel
  at System.Dynamic.DynamicMetaObjectBinder.Bind (System.Object[] args, System.Collections.ObjectModel.ReadOnlyCollection`1[T] parameters, System.Linq.Expressions.LabelTarget returnLabel) [0x001cd] in /persistent/mono/external/corefx/src/System.Linq.Expressions/src/System/Dynamic/DynamicMetaObjectBinder.cs:141 
  at System.Runtime.CompilerServices.CallSiteBinder.BindCore[T] (System.Runtime.CompilerServices.CallSite`1[T] site, System.Object[] args) [0x0001c] in /persistent/mono/external/corefx/src/System.Linq.Expressions/src/System/Runtime/CompilerServices/CallSiteBinder.cs:129 
  at System.Dynamic.UpdateDelegates.UpdateAndExecute2[T0,T1,TRet] (System.Runtime.CompilerServices.CallSite site, T0 arg0, T1 arg1) [0x00126] in /persistent/mono/external/corefx/src/System.Linq.Expressions/src/System/Dynamic/UpdateDelegates.Generated.cs:263 
  at (wrapper delegate-invoke) System.Func`4[System.Runtime.CompilerServices.CallSite,System.Object,System.Double,System.Object].invoke_TResult_T1_T2_T3(System.Runtime.CompilerServices.CallSite,object,double)
  at InverterScan.Program+<>c.<Main>b__46_0 () [0x00014] in C:\projects\Iridium\InverterScan\InverterScan\Program.cs:1199 
  at System.Threading.ThreadHelper.ThreadStart_Context (System.Object state) [0x00017] in /persistent/mono/mcs/class/referencesource/mscorlib/system/threading/thread.cs:74 
  at System.Threading.ExecutionContext.RunInternal (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, System.Boolean preserveSyncCtx) [0x0008d] in /persistent/mono/mcs/class/referencesource/mscorlib/system/threading/executioncontext.cs:968 
  at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, System.Boolean preserveSyncCtx) [0x00000] in /persistent/mono/mcs/class/referencesource/mscorlib/system/threading/executioncontext.cs:910 
  at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state) [0x00031] in /persistent/mono/mcs/class/referencesource/mscorlib/system/threading/executioncontext.cs:899 
  at System.Threading.ThreadHelper.ThreadStart () [0x0000b] in /persistent/mono/mcs/class/referencesource/mscorlib/system/threading/thread.cs:111 
[ERROR] FATAL UNHANDLED EXCEPTION: System.ArgumentNullException: Value cannot be null.
Parameter name: returnLabel
  at System.Dynamic.DynamicMetaObjectBinder.Bind (System.Object[] args, System.Collections.ObjectModel.ReadOnlyCollection`1[T] parameters, System.Linq.Expressions.LabelTarget returnLabel) [0x001cd] in /persistent/mono/external/corefx/src/System.Linq.Expressions/src/System/Dynamic/DynamicMetaObjectBinder.cs:141 
  at System.Runtime.CompilerServices.CallSiteBinder.BindCore[T] (System.Runtime.CompilerServices.CallSite`1[T] site, System.Object[] args) [0x0001c] in /persistent/mono/external/corefx/src/System.Linq.Expressions/src/System/Runtime/CompilerServices/CallSiteBinder.cs:129 
  at System.Dynamic.UpdateDelegates.UpdateAndExecute2[T0,T1,TRet] (System.Runtime.CompilerServices.CallSite site, T0 arg0, T1 arg1) [0x00126] in /persistent/mono/external/corefx/src/System.Linq.Expressions/src/System/Dynamic/UpdateDelegates.Generated.cs:263 
  at (wrapper delegate-invoke) System.Func`4[System.Runtime.CompilerServices.CallSite,System.Object,System.Double,System.Object].invoke_TResult_T1_T2_T3(System.Runtime.CompilerServices.CallSite,object,double)
  at InverterScan.Program+<>c.<Main>b__46_0 () [0x00014] in C:\projects\Iridium\InverterScan\InverterScan\Program.cs:1199 
  at System.Threading.ThreadHelper.ThreadStart_Context (System.Object state) [0x00017] in /persistent/mono/mcs/class/referencesource/mscorlib/system/threading/thread.cs:74 
  at System.Threading.ExecutionContext.RunInternal (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, System.Boolean preserveSyncCtx) [0x0008d] in /persistent/mono/mcs/class/referencesource/mscorlib/system/threading/executioncontext.cs:968 
  at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, System.Boolean preserveSyncCtx) [0x00000] in /persistent/mono/mcs/class/referencesource/mscorlib/system/threading/executioncontext.cs:910 
  at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state) [0x00031] in /persistent/mono/mcs/class/referencesource/mscorlib/system/threading/executioncontext.cs:899 
  at System.Threading.ThreadHelper.ThreadStart () [0x0000b] in /persistent/mono/mcs/class/referencesource/mscorlib/system/threading/thread.cs:111 
@jaykrell

This comment has been minimized.

Copy link
Collaborator

@jaykrell jaykrell commented Apr 17, 2019

Does it repro on x86, amd64, arm32, I am curious?
I'm running it on amd64. 100 iterations and no repro.
I should get back to arm32/arm64 some day.

@kdg3737

This comment has been minimized.

Copy link
Author

@kdg3737 kdg3737 commented Apr 17, 2019

I just tried it on x64/linux, mono 4.6.2, no errors at all.

@kdg3737

This comment has been minimized.

Copy link
Author

@kdg3737 kdg3737 commented Apr 17, 2019

Tried it on another arm64 device (Debian 9, mono 5.9.0), same result (crashes).

@jaykrell

This comment has been minimized.

Copy link
Collaborator

@jaykrell jaykrell commented Apr 19, 2019

I have arm64/windows/wsl hardware ordered, can try this when I get it.

@kdg3737

This comment has been minimized.

Copy link
Author

@kdg3737 kdg3737 commented Apr 23, 2019

Thanks for looking into this jaykrell. I ran it again and now have something that looks more like an actual stacktrace (the gdb must have kicked in for some reason), can't make much sense of it though:

> * Assertion at mini-arm64.c:886, condition `ji' not met
> 
> 
> =================================================================
> 	Native Crash Reporting
> =================================================================
> Got a SIGABRT while executing native code. This usually indicates
> a fatal error in the mono runtime or one of the native libraries 
> used by your application.
> =================================================================
> /proc/self/maps:
> 558843e000-55887ef000 r-xp 00000000 b3:07 4662                           /usr/local/bin/mono-sgen
> 55887fe000-5588806000 r--p 003b0000 b3:07 4662                           /usr/local/bin/mono-sgen
> 5588806000-558880c000 rw-p 003b8000 b3:07 4662                           /usr/local/bin/mono-sgen
> 558880c000-5588823000 rw-p 00000000 00:00 0 
> 55c491d000-55c4acc000 rw-p 00000000 00:00 0                              [heap]
> 7ef6600000-7ef6700000 rw-p 00000000 00:00 0 
> 7ef67f4000-7ef67f5000 ---p 00000000 00:00 0 
> 7ef67f5000-7ef69f5000 rw-p 00000000 00:00 0 
> 7ef69f5000-7ef69f6000 ---p 00000000 00:00 0 
> 7ef69f6000-7ef6bf6000 rw-p 00000000 00:00 0 
> 7ef6bf6000-7ef6bf7000 ---p 00000000 00:00 0 
> 7ef6bf7000-7ef6df7000 rw-p 00000000 00:00 0 
> 7ef6df7000-7ef6df8000 ---p 00000000 00:00 0 
> 7ef6df8000-7ef6ff8000 rw-p 00000000 00:00 0 
> 7ef6ff8000-7ef6ff9000 ---p 00000000 00:00 0 
> 7ef6ff9000-7ef71f9000 rw-p 00000000 00:00 0 
> 7ef71f9000-7ef71fa000 ---p 00000000 00:00 0 
> 7ef71fa000-7ef73fa000 rw-p 00000000 00:00 0 
> 7ef73fa000-7ef73fb000 ---p 00000000 00:00 0 
> 7ef73fb000-7ef75fb000 rw-p 00000000 00:00 0 
> 7ef75fb000-7ef75fc000 ---p 00000000 00:00 0 
> 7ef75fc000-7ef77fc000 rw-p 00000000 00:00 0 
> 7ef77fc000-7ef77fd000 ---p 00000000 00:00 0 
> 7ef77fd000-7ef79fd000 rw-p 00000000 00:00 0 
> 7ef79fd000-7ef79fe000 ---p 00000000 00:00 0 
> 
> =================================================================
> 	Native stacktrace:
> =================================================================
> 	0x55884e0184 - mono : (null)
> 
> =================================================================
> 	Telemetry Dumper:
> =================================================================
> Pkilling 0x7ef71f81d0 from 0x7f595ea1d0
> Pkilling 0x7f587e31d0 from 0x7f595ea1d0
> Pkilling 0x7f59dee1d0 from 0x7f595ea1d0
> Pkilling 0x7f783811d0 from 0x7f595ea1d0
> Pkilling 0x7f5b3f91d0 from 0x7f595ea1d0
> Pkilling 0x7ef73f91d0 from 0x7f595ea1d0
> Pkilling 0x7f589e41d0 from 0x7f595ea1d0
> Pkilling 0x7f59fef1d0 from 0x7f595ea1d0
> Pkilling 0x7f5b5fa1d0 from 0x7f595ea1d0
> Pkilling 0x7f785821d0 from 0x7f595ea1d0
> Pkilling 0x7ef75fa1d0 from 0x7f595ea1d0
> Pkilling 0x7f58be51d0 from 0x7f595ea1d0
> Pkilling 0x7f5a1f01d0 from 0x7f595ea1d0
> Pkilling 0x7f787831d0 from 0x7f595ea1d0
> Pkilling 0x7f5b7fb1d0 from 0x7f595ea1d0
> Pkilling 0x7ef77fb1d0 from 0x7f595ea1d0
> Pkilling 0x7f58de61d0 from 0x7f595ea1d0
> Pkilling 0x7f5a3f11d0 from 0x7f595ea1d0
> Pkilling 0x7f5b9fc1d0 from 0x7f595ea1d0
> Pkilling 0x7f789841d0 from 0x7f595ea1d0
> Pkilling 0x7f7cd10a40 from 0x7f595ea1d0
> Pkilling 0x7ef79fc1d0 from 0x7f595ea1d0
> Pkilling 0x7f58fe71d0 from 0x7f595ea1d0
> Pkilling 0x7f5a5f21d0 from 0x7f595ea1d0
> Pkilling 0x7f78b851d0 from 0x7f595ea1d0
> Pkilling 0x7f5bbfd1d0 from 0x7f595ea1d0
> Pkilling 0x7ef7bfd1d0 from 0x7f595ea1d0
> Pkilling 0x7f591e81d0 from 0x7f595ea1d0
> Pkilling 0x7f5a7f31d0 from 0x7f595ea1d0
> Pkilling 0x7f5bdfe1d0 from 0x7f595ea1d0
> Pkilling 0x7f78d861d0 from 0x7f595ea1d0
> Pkilling 0x7ef7dfe1d0 from 0x7f595ea1d0
> Pkilling 0x7f593e91d0 from 0x7f595ea1d0
> Pkilling 0x7f5a9f41d0 from 0x7f595ea1d0
> Pkilling 0x7f78f871d0 from 0x7f595ea1d0
> Pkilling 0x7f5bfff1d0 from 0x7f595ea1d0
> Pkilling 0x7ef69f41d0 from 0x7f595ea1d0
> Pkilling 0x7ef7fff1d0 from 0x7f595ea1d0
> Pkilling 0x7f5abf51d0 from 0x7f595ea1d0
> Pkilling 0x7ef6bf51d0 from 0x7f595ea1d0
> Pkilling 0x7f597eb1d0 from 0x7f595ea1d0
> Pkilling 0x7f5adf61d0 from 0x7f595ea1d0
> Pkilling 0x7ef6df61d0 from 0x7f595ea1d0
> Pkilling 0x7f583e11d0 from 0x7f595ea1d0
> Pkilling 0x7f599ec1d0 from 0x7f595ea1d0
> Pkilling 0x7f5aff71d0 from 0x7f595ea1d0
> Pkilling 0x7ef6ff71d0 from 0x7f595ea1d0
> Pkilling 0x7f585e21d0 from 0x7f595ea1d0
> Pkilling 0x7f59bed1d0 from 0x7f595ea1d0
> Pkilling 0x7f5b1f81d0 from 0x7f595ea1d0
> Pkilling 0x7f79eca1d0 from 0x7f595ea1d0
> Entering thread summarizer pause from 0x7f595ea1d0
> Finished thread summarizer pause from 0x7f595ea1d0.
> Trying to register response after dumping period endedThread 0x7f5a1f01d0 reported itself.
> Trying to register response after dumping period endedThread 0x7f78f871d0 reported itself.
> Trying to register response after dumping period endedThread 0x7f78d861d0 reported itself.
> Trying to register response after dumping period endedThread 0x7f5adf61d0 reported itself.
> Trying to register response after dumping period endedThread 0x7f5bdfe1d0 reported itself.
> Trying to register response after dumping period endedThread 0x7f5b9fc1d0 reported itself.
> Trying to register response after dumping period endedThread 0x7f785821d0 reported itself.
> Trying to register response after dumping period endedThread 0x7f789841d0 reported itself.
> Trying to register response after dumping period endedThread 0x7f5bbfd1d0 reported itself.
> Trying to register response after dumping period endedThread 0x7f787831d0 reported itself.
> Trying to register response after dumping period endedThread 0x7f5b5fa1d0 reported itself.
> Trying to register response after dumping period endedThread 0x7f5a3f11d0 reported itself.
> Trying to register response after dumping period endedThread 0x7f5abf51d0 reported itself.
> Trying to register response after dumping period endedThread 0x7f5a7f31d0 reported itself.
> Trying to register response after dumping period endedThread 0x7f5b7fb1d0 reported itself.
> Trying to register response after dumping period endedThread 0x7f5a5f21d0 reported itself.
> Trying to register response after dumping period endedThread 0x7f5bfff1d0 reported itself.
> Trying to register response after dumping period endedTrying to register response after dumping period endedThread 0x7f78b851d0 reported itself.
> Thread 0x7f5aff71d0 reported itself.
> Trying to register response after dumping period endedThread 0x7f5a9f41d0 reported itself.
> Trying to register response after dumping period endedThread 0x7f5b1f81d0 reported itself.
> Trying to register response after dumping period endedThread 0x7f7cd10a40 reported itself.
> 
> Waiting for dumping threads to resume
> 
> =================================================================
> 	External Debugger Dump:
> =================================================================
> [New LWP 8847]
> [New LWP 8848]
> [New LWP 8849]
> [New LWP 8850]
> [New LWP 8851]
> [New LWP 8852]
> [New LWP 8853]
> [New LWP 8854]
> [New LWP 8856]
> [New LWP 8857]
> [New LWP 8858]
> [New LWP 8859]
> [New LWP 8860]
> [New LWP 8861]
> [New LWP 8863]
> [New LWP 8864]
> [New LWP 8865]
> [New LWP 8866]
> [New LWP 8867]
> [New LWP 8868]
> [New LWP 8869]
> [New LWP 8870]
> [New LWP 8871]
> [New LWP 8877]
> warning: File "/usr/local/bin/mono-sgen-gdb.py" auto-loading has been declined by your `auto-load safe-path' set to "$debugdir:$datadir/auto-load".
> To enable execution of this file add
> 	add-auto-load-safe-path /usr/local/bin/mono-sgen-gdb.py
> line to your configuration file "/home/debian/.gdbinit".
> To completely disable this security protection add
> 	set auto-load safe-path /
> line to your configuration file "/home/debian/.gdbinit".
> For more information about this security protection see the
> "Auto-loading safe path" section in the GDB manual.  E.g., run from the shell:
> 	info "(gdb)Auto-loading safe path"
> [Thread debugging using libthread_db enabled]
> Using host libthread_db library "/lib/aarch64-linux-gnu/libthread_db.so.1".
> 0x0000007f7cbbdb1c in do_futex_wait () from /lib/aarch64-linux-gnu/libpthread.so.0
>   Id   Target Id         Frame 
> * 1    Thread 0x7f7cd10a40 (LWP 8846) "mono" 0x0000007f7cbbdb1c in do_futex_wait () from /lib/aarch64-linux-gnu/libpthread.so.0
>   2    Thread 0x7f7c3ff1d0 (LWP 8847) "SGen worker" 0x0000007f7cbbb33c in pthread_cond_wait@@GLIBC_2.17 () from /lib/aarch64-linux-gnu/libpthread.so.0
>   3    Thread 0x7f79eca1d0 (LWP 8848) "Finalizer" 0x0000007f7cbbd8e0 in do_futex_wait.constprop () from /lib/aarch64-linux-gnu/libpthread.so.0
>   4    Thread 0x7f78f871d0 (LWP 8849) "mono" 0x0000007f7cbbdb1c in do_futex_wait () from /lib/aarch64-linux-gnu/libpthread.so.0
>   5    Thread 0x7f78d861d0 (LWP 8850) "mono" 0x0000007f7cbbdb1c in do_futex_wait () from /lib/aarch64-linux-gnu/libpthread.so.0
>   6    Thread 0x7f78b851d0 (LWP 8851) "mono" 0x0000007f7cbbdb1c in do_futex_wait () from /lib/aarch64-linux-gnu/libpthread.so.0
>   7    Thread 0x7f789841d0 (LWP 8852) "mono" 0x0000007f7cbbdb1c in do_futex_wait () from /lib/aarch64-linux-gnu/libpthread.so.0
>   8    Thread 0x7f787831d0 (LWP 8853) "mono" 0x0000007f7cbbdb1c in do_futex_wait () from /lib/aarch64-linux-gnu/libpthread.so.0
>   9    Thread 0x7f785821d0 (LWP 8854) "mono" 0x0000007f7cbbdb1c in do_futex_wait () from /lib/aarch64-linux-gnu/libpthread.so.0
>   10   Thread 0x7f5bfff1d0 (LWP 8856) "mono" 0x0000007f7cbbdb1c in do_futex_wait () from /lib/aarch64-linux-gnu/libpthread.so.0
>   11   Thread 0x7f5bdfe1d0 (LWP 8857) "mono" 0x0000007f7cbbdb1c in do_futex_wait () from /lib/aarch64-linux-gnu/libpthread.so.0
>   12   Thread 0x7f5bbfd1d0 (LWP 8858) "mono" 0x0000007f7cbbdb1c in do_futex_wait () from /lib/aarch64-linux-gnu/libpthread.so.0
>   13   Thread 0x7f5b9fc1d0 (LWP 8859) "mono" 0x0000007f7cbbdb1c in do_futex_wait () from /lib/aarch64-linux-gnu/libpthread.so.0
>   14   Thread 0x7f5b7fb1d0 (LWP 8860) "mono" 0x0000007f7cbbdb1c in do_futex_wait () from /lib/aarch64-linux-gnu/libpthread.so.0
>   15   Thread 0x7f5b5fa1d0 (LWP 8861) "mono" 0x0000007f7cbbdb1c in do_futex_wait () from /lib/aarch64-linux-gnu/libpthread.so.0
>   16   Thread 0x7f5b1f81d0 (LWP 8863) "mono" 0x0000007f7cbbdb1c in do_futex_wait () from /lib/aarch64-linux-gnu/libpthread.so.0
>   17   Thread 0x7f5aff71d0 (LWP 8864) "mono" 0x0000007f7cbbdb1c in do_futex_wait () from /lib/aarch64-linux-gnu/libpthread.so.0
>   18   Thread 0x7f5adf61d0 (LWP 8865) "mono" 0x0000007f7cbbdb1c in do_futex_wait () from /lib/aarch64-linux-gnu/libpthread.so.0
>   19   Thread 0x7f5abf51d0 (LWP 8866) "mono" 0x0000007f7cbbdb1c in do_futex_wait () from /lib/aarch64-linux-gnu/libpthread.so.0
>   20   Thread 0x7f5a9f41d0 (LWP 8867) "mono" 0x0000007f7cbbdb1c in do_futex_wait () from /lib/aarch64-linux-gnu/libpthread.so.0
>   21   Thread 0x7f5a7f31d0 (LWP 8868) "mono" 0x0000007f7cbbdb1c in do_futex_wait () from /lib/aarch64-linux-gnu/libpthread.so.0
>   22   Thread 0x7f5a5f21d0 (LWP 8869) "mono" 0x0000007f7cbbdb1c in do_futex_wait () from /lib/aarch64-linux-gnu/libpthread.so.0
>   23   Thread 0x7f5a3f11d0 (LWP 8870) "mono" 0x0000007f7cbbdb1c in do_futex_wait () from /lib/aarch64-linux-gnu/libpthread.so.0
>   24   Thread 0x7f5a1f01d0 (LWP 8871) "mono" 0x0000007f7cbbdb1c in do_futex_wait () from /lib/aarch64-linux-gnu/libpthread.so.0
>   25   Thread 0x7f595ea1d0 (LWP 8877) "mono" 0x0000007f7cbbfaa8 in waitpid () from /lib/aarch64-linux-gnu/libpthread.so.0
> 
> Thread 25 (Thread 0x7f595ea1d0 (LWP 8877)):
> #0  0x0000007f7cbbfaa8 in waitpid () from /lib/aarch64-linux-gnu/libpthread.so.0
> #1  0x00000055884e0368 in dump_native_stacktrace (mctx=0x7f7caaee38 <fprintf+104>, mctx@entry=0x7f595e7450, signal=0x5588735e20 "SIGABRT") at mini-posix.c:1111
> #2  0x00000055884e045c in mono_dump_native_crash_info (signal=signal@entry=0x5588735e20 "SIGABRT", mctx=mctx@entry=0x7f595e7450, info=info@entry=0x7f595e7760) at mini-posix.c:1153
> #3  0x00000055884b392c in mono_handle_native_crash (signal=signal@entry=0x5588735e20 "SIGABRT", mctx=mctx@entry=0x7f595e7450, info=info@entry=0x7f595e7760) at mini-exceptions.c:3326
> #4  0x00000055884df730 in sigabrt_signal_handler (_dummy=6, _info=0x7f595e7760, context=<optimized out>) at mini-posix.c:234
> #5  <signal handler called>
> #6  0x0000007f7ca949fc in raise () from /lib/aarch64-linux-gnu/libc.so.6
> #7  0x0000007f7ca95df4 in abort () from /lib/aarch64-linux-gnu/libc.so.6
> #8  0x000000558871a66c in monoeg_assert_abort () at goutput.c:57
> #9  0x0000005588700a60 in mono_log_write_logfile (log_domain=<optimized out>, level=G_LOG_LEVEL_ERROR, hdr=<optimized out>, message=0x7ef801c880 "* Assertion at mini-arm64.c:886, condition `ji' not met\n") at mono-log-common.c:136
> #10 0x000000558871a5d0 in monoeg_g_logstr (msg=<optimized out>, log_level=G_LOG_LEVEL_ERROR, log_domain=0x0) at goutput.c:134
> #11 monoeg_g_logv_nofree (log_domain=log_domain@entry=0x0, log_level=log_level@entry=G_LOG_LEVEL_ERROR, format=format@entry=0x55887234e0 "* Assertion at %s:%d, condition `%s' not met\n", args=<error reading variable: Cannot access memory at address 0x8>) at goutput.c:149
> #12 0x000000558871a9f0 in monoeg_assertion_message (format=format@entry=0x55887234e0 "* Assertion at %s:%d, condition `%s' not met\n") at goutput.c:184
> #13 0x00000055884ce72c in create_thunk (cfg=0x0, domain=0x55c492bee0, code=code@entry=0x7f581d960c "2", target=0x7f7800dc18 "\375{\274\251\375\003") at mini-arm64.c:886
> #14 0x00000055884cfc70 in arm_patch_full (cfg=<optimized out>, domain=<optimized out>, code=0x7f581d960c "2", target=<optimized out>, relocation=<optimized out>) at mini-arm64.c:941
> #15 0x000000558847cd58 in mini_patch_jump_sites (domain=domain@entry=0x55c492bee0, method=<optimized out>, method@entry=0x7f6c01bee0, addr=0x7f7800dc18) at mini-runtime.c:1754
> #16 0x00000055884b7a78 in common_call_trampoline (regs=regs@entry=0x7f595e9100, code=code@entry=0x0, m=m@entry=0x7f6c01bee0, vt=vt@entry=0x0, vtable_slot=<optimized out>, vtable_slot@entry=0x0, error=error@entry=0x7f595e9088) at mini-trampolines.c:662
> #17 0x00000055884b7ccc in mono_magic_trampoline (regs=0x7f595e9100, code=0x0, arg=0x7f6c01bee0, tramp=<optimized out>) at mini-trampolines.c:771
> #18 0x0000007f7ccdf5fc in ?? ()
> #19 0x0000007f7c40ef30 in ?? ()
> #20 0x0000007f7c94ffd8 in ?? ()
> Backtrace stopped: previous frame inner to this frame (corrupt stack?)
> 
> Thread 24 (Thread 0x7f5a1f01d0 (LWP 8871)):
> #0  0x0000007f7cbbdb1c in do_futex_wait () from /lib/aarch64-linux-gnu/libpthread.so.0
> #1  0x0000007f7cbbdbe8 in __new_sem_wait_slow () from /lib/aarch64-linux-gnu/libpthread.so.0
> #2  0x0000007f7cbbdd04 in sem_timedwait () from /lib/aarch64-linux-gnu/libpthread.so.0
> #3  0x0000005588651f08 in mono_os_sem_timedwait (flags=MONO_SEM_FLAGS_NONE, timeout_ms=1000, sem=0x7ef644a008) at ../../mono/utils/mono-os-semaphore.h:252
> #4  summarizer_state_wait (thread=<optimized out>) at threads.c:6522
> #5  mono_threads_summarize_execute (ctx=0x55888218e0 <suspend_signal_mask>, ctx@entry=0x7f5a1edce0, out=0x7f100008c0, out@entry=0x7f5a1edce8, hashes=0x7f5a1edcd0, hashes@entry=0x7f5a1edcf0, silent=silent@entry=0, working_mem=0x558871234c <suspend_signal_handler+228> "`\242C\371 \002", working_mem@entry=0x0, provided_size=546972818560, provided_size@entry=0) at threads.c:6584
> #6  0x00000055884df650 in sigterm_signal_handler (_dummy=15, _info=0x7f5a1edff0, context=0x7f5a1ee070) at mini-posix.c:251
> #7  <signal handler called>
> #8  0x0000007f7cbbe5e0 in __lll_lock_wait () from /lib/aarch64-linux-gnu/libpthread.so.0
> #9  0x0000007f7cbb797c in pthread_mutex_lock () from /lib/aarch64-linux-gnu/libpthread.so.0
> #10 0x00000055886cb720 in mono_os_mutex_lock (mutex=0x55888227c8 <sgen_gc_mutex>) at ../../mono/utils/mono-os-mutex.h:103
> #11 mono_coop_mutex_lock (mutex=0x55888227c8 <sgen_gc_mutex>) at ../../mono/utils/mono-coop-mutex.h:54
> #12 sgen_gc_lock () at sgen-gc.c:3766
> #13 0x00000055886bcc6c in sgen_alloc_obj (vtable=0x7f00048ac8, size=24) at sgen-alloc.c:423
> #14 0x00000055886a2460 in mono_gc_alloc_obj (vtable=<optimized out>, size=<optimized out>) at sgen-mono.c:917
> #15 0x0000007f7c9c148c in ?? ()
> #16 0x0000007f5a1ef2c0 in ?? ()
> Backtrace stopped: previous frame inner to this frame (corrupt stack?)
> 
> Thread 23 (Thread 0x7f5a3f11d0 (LWP 8870)):
> #0  0x0000007f7cbbdb1c in do_futex_wait () from /lib/aarch64-linux-gnu/libpthread.so.0
> #1  0x0000007f7cbbdbe8 in __new_sem_wait_slow () from /lib/aarch64-linux-gnu/libpthread.so.0
> #2  0x0000007f7cbbdd04 in sem_timedwait () from /lib/aarch64-linux-gnu/libpthread.so.0
> #3  0x0000005588651f08 in mono_os_sem_timedwait (flags=MONO_SEM_FLAGS_NONE, timeout_ms=1000, sem=0x7ef62dd008) at ../../mono/utils/mono-os-semaphore.h:252
> #4  summarizer_state_wait (thread=<optimized out>) at threads.c:6522
> #5  mono_threads_summarize_execute (ctx=0x55888218e0 <suspend_signal_mask>, ctx@entry=0x7f5a3eeba0, out=0x7f1c0008c0, out@entry=0x7f5a3eeba8, hashes=0x7f5a3eeb90, hashes@entry=0x7f5a3eebb0, silent=silent@entry=0, working_mem=0x558871234c <suspend_signal_handler+228> "`\242C\371 \002", working_mem@entry=0x0, provided_size=546974919488, provided_size@entry=0) at threads.c:6584
> #6  0x00000055884df650 in sigterm_signal_handler (_dummy=15, _info=0x7f5a3eeeb0, context=0x7f5a3eef30) at mini-posix.c:251
> #7  <signal handler called>
> #8  0x0000007f7cbbe5e0 in __lll_lock_wait () from /lib/aarch64-linux-gnu/libpthread.so.0
> #9  0x0000007f7cbb797c in pthread_mutex_lock () from /lib/aarch64-linux-gnu/libpthread.so.0
> #10 0x00000055886cb720 in mono_os_mutex_lock (mutex=0x55888227c8 <sgen_gc_mutex>) at ../../mono/utils/mono-os-mutex.h:103
> #11 mono_coop_mutex_lock (mutex=0x55888227c8 <sgen_gc_mutex>) at ../../mono/utils/mono-coop-mutex.h:54
> #12 sgen_gc_lock () at sgen-gc.c:3766
> #13 0x00000055886a2554 in mono_gc_alloc_vector (vtable=0x55c49def28, size=96, max_length=8) at sgen-mono.c:1316
> #14 0x0000007f7c9c3ac4 in ?? ()
> #15 0x0000007f1c002df0 in ?? ()
> Backtrace stopped: previous frame inner to this frame (corrupt stack?)
> 
> Thread 22 (Thread 0x7f5a5f21d0 (LWP 8869)):
> #0  0x0000007f7cbbdb1c in do_futex_wait () from /lib/aarch64-linux-gnu/libpthread.so.0
> #1  0x0000007f7cbbdbe8 in __new_sem_wait_slow () from /lib/aarch64-linux-gnu/libpthread.so.0
> #2  0x0000007f7cbbdd04 in sem_timedwait () from /lib/aarch64-linux-gnu/libpthread.so.0
> #3  0x0000005588651f08 in mono_os_sem_timedwait (flags=MONO_SEM_FLAGS_NONE, timeout_ms=1000, sem=0x7ef5e4d008) at ../../mono/utils/mono-os-semaphore.h:252
> #4  summarizer_state_wait (thread=<optimized out>) at threads.c:6522
> #5  mono_threads_summarize_execute (ctx=0x55888218e0 <suspend_signal_mask>, ctx@entry=0x7f5a5efba0, out=0x7f180008c0, out@entry=0x7f5a5efba8, hashes=0x7f5a5efb90, hashes@entry=0x7f5a5efbb0, silent=silent@entry=0, working_mem=0x558871234c <suspend_signal_handler+228> "`\242C\371 \002", working_mem@entry=0x0, provided_size=546977020736, provided_size@entry=0) at threads.c:6584
> #6  0x00000055884df650 in sigterm_signal_handler (_dummy=15, _info=0x7f5a5efeb0, context=0x7f5a5eff30) at mini-posix.c:251
> #7  <signal handler called>
> #8  0x0000007f7cbbe5e0 in __lll_lock_wait () from /lib/aarch64-linux-gnu/libpthread.so.0
> #9  0x0000007f7cbb797c in pthread_mutex_lock () from /lib/aarch64-linux-gnu/libpthread.so.0
> #10 0x00000055886cb720 in mono_os_mutex_lock (mutex=0x55888227c8 <sgen_gc_mutex>) at ../../mono/utils/mono-os-mutex.h:103
> #11 mono_coop_mutex_lock (mutex=0x55888227c8 <sgen_gc_mutex>) at ../../mono/utils/mono-coop-mutex.h:54
> #12 sgen_gc_lock () at sgen-gc.c:3766
> #13 0x00000055886a2554 in mono_gc_alloc_vector (vtable=0x55c49def28, size=96, max_length=8) at sgen-mono.c:1316
> #14 0x0000007f7c9c3ac4 in ?? ()
> #15 0x0000007f18002df0 in ?? ()
> Backtrace stopped: previous frame inner to this frame (corrupt stack?)
> 
> Thread 21 (Thread 0x7f5a7f31d0 (LWP 8868)):
> #0  0x0000007f7cbbdb1c in do_futex_wait () from /lib/aarch64-linux-gnu/libpthread.so.0
> #1  0x0000007f7cbbdbe8 in __new_sem_wait_slow () from /lib/aarch64-linux-gnu/libpthread.so.0
> #2  0x0000007f7cbbdd04 in sem_timedwait () from /lib/aarch64-linux-gnu/libpthread.so.0
> #3  0x0000005588651f08 in mono_os_sem_timedwait (flags=MONO_SEM_FLAGS_NONE, timeout_ms=1000, sem=0x7ef604c008) at ../../mono/utils/mono-os-semaphore.h:252
> #4  summarizer_state_wait (thread=<optimized out>) at threads.c:6522
> #5  mono_threads_summarize_execute (ctx=0x55888218e0 <suspend_signal_mask>, ctx@entry=0x7f5a7f0ba0, out=0x7f240008c0, out@entry=0x7f5a7f0ba8, hashes=0x7f5a7f0b90, hashes@entry=0x7f5a7f0bb0, silent=silent@entry=0, working_mem=0x558871234c <suspend_signal_handler+228> "`\242C\371 \002", working_mem@entry=0x0, provided_size=546979121984, provided_size@entry=0) at threads.c:6584
> #6  0x00000055884df650 in sigterm_signal_handler (_dummy=15, _info=0x7f5a7f0eb0, context=0x7f5a7f0f30) at mini-posix.c:251
> #7  <signal handler called>
> #8  0x0000007f7cbbe5b0 in __lll_lock_wait () from /lib/aarch64-linux-gnu/libpthread.so.0
> #9  0x0000007f7cbb797c in pthread_mutex_lock () from /lib/aarch64-linux-gnu/libpthread.so.0
> #10 0x00000055886cb720 in mono_os_mutex_lock (mutex=0x55888227c8 <sgen_gc_mutex>) at ../../mono/utils/mono-os-mutex.h:103
> #11 mono_coop_mutex_lock (mutex=0x55888227c8 <sgen_gc_mutex>) at ../../mono/utils/mono-coop-mutex.h:54
> #12 sgen_gc_lock () at sgen-gc.c:3766
> #13 0x00000055886a2554 in mono_gc_alloc_vector (vtable=0x55c49def28, size=96, max_length=8) at sgen-mono.c:1316
> #14 0x0000007f7c9c3ac4 in ?? ()
> #15 0x0000007f24002df0 in ?? ()
> Backtrace stopped: previous frame inner to this frame (corrupt stack?)
> 
> Thread 20 (Thread 0x7f5a9f41d0 (LWP 8867)):
> #0  0x0000007f7cbbdb1c in do_futex_wait () from /lib/aarch64-linux-gnu/libpthread.so.0
> #1  0x0000007f7cbbdbe8 in __new_sem_wait_slow () from /lib/aarch64-linux-gnu/libpthread.so.0
> #2  0x0000007f7cbbdd04 in sem_timedwait () from /lib/aarch64-linux-gnu/libpthread.so.0
> #3  0x0000005588651f08 in mono_os_sem_timedwait (flags=MONO_SEM_FLAGS_NONE, timeout_ms=1000, sem=0x7f58054008) at ../../mono/utils/mono-os-semaphore.h:252
> #4  summarizer_state_wait (thread=<optimized out>) at threads.c:6522
> #5  mono_threads_summarize_execute (ctx=0x55888218e0 <suspend_signal_mask>, ctx@entry=0x7f5a9f1ce0, out=0x7f200008c0, out@entry=0x7f5a9f1ce8, hashes=0x7f5a9f1cd0, hashes@entry=0x7f5a9f1cf0, silent=silent@entry=0, working_mem=0x558871234c <suspend_signal_handler+228> "`\242C\371 \002", working_mem@entry=0x0, provided_size=546981223552, provided_size@entry=0) at threads.c:6584
> #6  0x00000055884df650 in sigterm_signal_handler (_dummy=15, _info=0x7f5a9f1ff0, context=0x7f5a9f2070) at mini-posix.c:251
> #7  <signal handler called>
> #8  0x0000007f7cbbe5dc in __lll_lock_wait () from /lib/aarch64-linux-gnu/libpthread.so.0
> #9  0x0000007f7cbb797c in pthread_mutex_lock () from /lib/aarch64-linux-gnu/libpthread.so.0
> #10 0x00000055886cb720 in mono_os_mutex_lock (mutex=0x55888227c8 <sgen_gc_mutex>) at ../../mono/utils/mono-os-mutex.h:103
> #11 mono_coop_mutex_lock (mutex=0x55888227c8 <sgen_gc_mutex>) at ../../mono/utils/mono-coop-mutex.h:54
> #12 sgen_gc_lock () at sgen-gc.c:3766
> #13 0x00000055886bcc6c in sgen_alloc_obj (vtable=0x7f00048ac8, size=24) at sgen-alloc.c:423
> #14 0x00000055886a2460 in mono_gc_alloc_obj (vtable=<optimized out>, size=<optimized out>) at sgen-mono.c:917
> #15 0x0000007f7c9c148c in ?? ()
> #16 0x0000007f5a9f32c0 in ?? ()
> Backtrace stopped: previous frame inner to this frame (corrupt stack?)
> 
> Thread 19 (Thread 0x7f5abf51d0 (LWP 8866)):
> #0  0x0000007f7cbbdb1c in do_futex_wait () from /lib/aarch64-linux-gnu/libpthread.so.0
> #1  0x0000007f7cbbdbe8 in __new_sem_wait_slow () from /lib/aarch64-linux-gnu/libpthread.so.0
> #2  0x0000007f7cbbdd04 in sem_timedwait () from /lib/aarch64-linux-gnu/libpthread.so.0
> #3  0x0000005588651f08 in mono_os_sem_timedwait (flags=MONO_SEM_FLAGS_NONE, timeout_ms=1000, sem=0x7ef64dc008) at ../../mono/utils/mono-os-semaphore.h:252
> #4  summarizer_state_wait (thread=<optimized out>) at threads.c:6522
> #5  mono_threads_summarize_execute (ctx=0x55888218e0 <suspend_signal_mask>, ctx@entry=0x7f5abf2ba0, out=0x7f2c0008c0, out@entry=0x7f5abf2ba8, hashes=0x7f5abf2b90, hashes@entry=0x7f5abf2bb0, silent=silent@entry=0, working_mem=0x558871234c <suspend_signal_handler+228> "`\242C\371 \002", working_mem@entry=0x0, provided_size=546983324480, provided_size@entry=0) at threads.c:6584
> #6  0x00000055884df650 in sigterm_signal_handler (_dummy=15, _info=0x7f5abf2eb0, context=0x7f5abf2f30) at mini-posix.c:251
> #7  <signal handler called>
> #8  0x0000007f7cbbe5dc in __lll_lock_wait () from /lib/aarch64-linux-gnu/libpthread.so.0
> #9  0x0000007f7cbb797c in pthread_mutex_lock () from /lib/aarch64-linux-gnu/libpthread.so.0
> #10 0x00000055886cb720 in mono_os_mutex_lock (mutex=0x55888227c8 <sgen_gc_mutex>) at ../../mono/utils/mono-os-mutex.h:103
> #11 mono_coop_mutex_lock (mutex=0x55888227c8 <sgen_gc_mutex>) at ../../mono/utils/mono-coop-mutex.h:54
> #12 sgen_gc_lock () at sgen-gc.c:3766
> #13 0x00000055886a2554 in mono_gc_alloc_vector (vtable=0x55c49def28, size=96, max_length=8) at sgen-mono.c:1316
> #14 0x0000007f7c9c3ac4 in ?? ()
> #15 0x0000007f2c002df0 in ?? ()
> Backtrace stopped: previous frame inner to this frame (corrupt stack?)
> 
> Thread 18 (Thread 0x7f5adf61d0 (LWP 8865)):
> #0  0x0000007f7cbbdb1c in do_futex_wait () from /lib/aarch64-linux-gnu/libpthread.so.0
> #1  0x0000007f7cbbdbe8 in __new_sem_wait_slow () from /lib/aarch64-linux-gnu/libpthread.so.0
> #2  0x0000007f7cbbdd04 in sem_timedwait () from /lib/aarch64-linux-gnu/libpthread.so.0
> #3  0x0000005588651f08 in mono_os_sem_timedwait (flags=MONO_SEM_FLAGS_NONE, timeout_ms=1000, sem=0x7ef6762008) at ../../mono/utils/mono-os-semaphore.h:252
> #4  summarizer_state_wait (thread=<optimized out>) at threads.c:6522
> #5  mono_threads_summarize_execute (ctx=0x55888218e0 <suspend_signal_mask>, ctx@entry=0x7f5adf3bb0, out=0x7f280008c0, out@entry=0x7f5adf3bb8, hashes=0x7f5adf3ba0, hashes@entry=0x7f5adf3bc0, silent=silent@entry=0, working_mem=0x558871234c <suspend_signal_handler+228> "`\242C\371 \002", working_mem@entry=0x0, provided_size=546985425744, provided_size@entry=0) at threads.c:6584
> #6  0x00000055884df650 in sigterm_signal_handler (_dummy=15, _info=0x7f5adf3ec0, context=0x7f5adf3f40) at mini-posix.c:251
> #7  <signal handler called>
> #8  0x0000007f7cbbe5dc in __lll_lock_wait () from /lib/aarch64-linux-gnu/libpthread.so.0
> #9  0x0000007f7cbb797c in pthread_mutex_lock () from /lib/aarch64-linux-gnu/libpthread.so.0
> #10 0x00000055886cb720 in mono_os_mutex_lock (mutex=0x55888227c8 <sgen_gc_mutex>) at ../../mono/utils/mono-os-mutex.h:103
> #11 mono_coop_mutex_lock (mutex=0x55888227c8 <sgen_gc_mutex>) at ../../mono/utils/mono-coop-mutex.h:54
> #12 sgen_gc_lock () at sgen-gc.c:3766
> #13 0x00000055886bcc6c in sgen_alloc_obj (vtable=0x55c4a088c8, size=40) at sgen-alloc.c:423
> #14 0x00000055886a2460 in mono_gc_alloc_obj (vtable=<optimized out>, size=<optimized out>) at sgen-mono.c:917
> #15 0x0000007f7c9c148c in ?? ()
> #16 0x0000007f5adf5950 in ?? ()
> Backtrace stopped: previous frame inner to this frame (corrupt stack?)
> 
> Thread 17 (Thread 0x7f5aff71d0 (LWP 8864)):
> #0  0x0000007f7cbbdb1c in do_futex_wait () from /lib/aarch64-linux-gnu/libpthread.so.0
> #1  0x0000007f7cbbdbe8 in __new_sem_wait_slow () from /lib/aarch64-linux-gnu/libpthread.so.0
> #2  0x0000007f7cbbdd04 in sem_timedwait () from /lib/aarch64-linux-gnu/libpthread.so.0
> #3  0x0000005588651f08 in mono_os_sem_timedwait (flags=MONO_SEM_FLAGS_NONE, timeout_ms=1000, sem=0x7ef656e008) at ../../mono/utils/mono-os-semaphore.h:252
> #4  summarizer_state_wait (thread=<optimized out>) at threads.c:6522
> #5  mono_threads_summarize_execute (ctx=0x55888218e0 <suspend_signal_mask>, ctx@entry=0x7f5aff4ba0, out=0x7f340008c0, out@entry=0x7f5aff4ba8, hashes=0x7f5aff4b90, hashes@entry=0x7f5aff4bb0, silent=silent@entry=0, working_mem=0x558871234c <suspend_signal_handler+228> "`\242C\371 \002", working_mem@entry=0x0, provided_size=546987526976, provided_size@entry=0) at threads.c:6584
> #6  0x00000055884df650 in sigterm_signal_handler (_dummy=15, _info=0x7f5aff4eb0, context=0x7f5aff4f30) at mini-posix.c:251
> #7  <signal handler called>
> #8  0x0000007f7cbbe5ac in __lll_lock_wait () from /lib/aarch64-linux-gnu/libpthread.so.0
> #9  0x0000007f7cbb797c in pthread_mutex_lock () from /lib/aarch64-linux-gnu/libpthread.so.0
> #10 0x00000055886cb720 in mono_os_mutex_lock (mutex=0x55888227c8 <sgen_gc_mutex>) at ../../mono/utils/mono-os-mutex.h:103
> #11 mono_coop_mutex_lock (mutex=0x55888227c8 <sgen_gc_mutex>) at ../../mono/utils/mono-coop-mutex.h:54
> #12 sgen_gc_lock () at sgen-gc.c:3766
> #13 0x00000055886a2554 in mono_gc_alloc_vector (vtable=0x55c49def28, size=96, max_length=8) at sgen-mono.c:1316
> #14 0x0000007f7c9c3ac4 in ?? ()
> #15 0x0000007f34002df0 in ?? ()
> Backtrace stopped: previous frame inner to this frame (corrupt stack?)
> 
> Thread 16 (Thread 0x7f5b1f81d0 (LWP 8863)):
> #0  0x0000007f7cbbdb1c in do_futex_wait () from /lib/aarch64-linux-gnu/libpthread.so.0
> #1  0x0000007f7cbbdbe8 in __new_sem_wait_slow () from /lib/aarch64-linux-gnu/libpthread.so.0
> #2  0x0000007f7cbbdd04 in sem_timedwait () from /lib/aarch64-linux-gnu/libpthread.so.0
> #3  0x0000005588651f08 in mono_os_sem_timedwait (flags=MONO_SEM_FLAGS_NONE, timeout_ms=1000, sem=0x7ef6493008) at ../../mono/utils/mono-os-semaphore.h:252
> #4  summarizer_state_wait (thread=<optimized out>) at threads.c:6522
> #5  mono_threads_summarize_execute (ctx=0x55888218e0 <suspend_signal_mask>, ctx@entry=0x7f5b1f5ba0, out=0x7f300008c0, out@entry=0x7f5b1f5ba8, hashes=0x7f5b1f5b90, hashes@entry=0x7f5b1f5bb0, silent=silent@entry=0, working_mem=0x558871234c <suspend_signal_handler+228> "`\242C\371 \002", working_mem@entry=0x0, provided_size=546989628224, provided_size@entry=0) at threads.c:6584
> #6  0x00000055884df650 in sigterm_signal_handler (_dummy=15, _info=0x7f5b1f5eb0, context=0x7f5b1f5f30) at mini-posix.c:251
> #7  <signal handler called>
> #8  0x0000007f7cbbe5dc in __lll_lock_wait () from /lib/aarch64-linux-gnu/libpthread.so.0
> #9  0x0000007f7cbb797c in pthread_mutex_lock () from /lib/aarch64-linux-gnu/libpthread.so.0
> #10 0x00000055886cb720 in mono_os_mutex_lock (mutex=0x55888227c8 <sgen_gc_mutex>) at ../../mono/utils/mono-os-mutex.h:103
> #11 mono_coop_mutex_lock (mutex=0x55888227c8 <sgen_gc_mutex>) at ../../mono/utils/mono-coop-mutex.h:54
> #12 sgen_gc_lock () at sgen-gc.c:3766
> #13 0x00000055886a2554 in mono_gc_alloc_vector (vtable=0x55c49def28, size=96, max_length=8) at sgen-mono.c:1316
> #14 0x0000007f7c9c3ac4 in ?? ()
> #15 0x0000007f30002df0 in ?? ()
> Backtrace stopped: previous frame inner to this frame (corrupt stack?)
> 
> Thread 15 (Thread 0x7f5b5fa1d0 (LWP 8861)):
> #0  0x0000007f7cbbdb1c in do_futex_wait () from /lib/aarch64-linux-gnu/libpthread.so.0
> #1  0x0000007f7cbbdbe8 in __new_sem_wait_slow () from /lib/aarch64-linux-gnu/libpthread.so.0
> #2  0x0000007f7cbbdd04 in sem_timedwait () from /lib/aarch64-linux-gnu/libpthread.so.0
> #3  0x0000005588651f08 in mono_os_sem_timedwait (flags=MONO_SEM_FLAGS_NONE, timeout_ms=1000, sem=0x7ef6294008) at ../../mono/utils/mono-os-semaphore.h:252
> #4  summarizer_state_wait (thread=<optimized out>) at threads.c:6522
> #5  mono_threads_summarize_execute (ctx=0x55888218e0 <suspend_signal_mask>, ctx@entry=0x7f5b5f7bb0, out=0x7f380008c0, out@entry=0x7f5b5f7bb8, hashes=0x7f5b5f7ba0, hashes@entry=0x7f5b5f7bc0, silent=silent@entry=0, working_mem=0x558871234c <suspend_signal_handler+228> "`\242C\371 \002", working_mem@entry=0x0, provided_size=546993830736, provided_size@entry=0) at threads.c:6584
> #6  0x00000055884df650 in sigterm_signal_handler (_dummy=15, _info=0x7f5b5f7ec0, context=0x7f5b5f7f40) at mini-posix.c:251
> #7  <signal handler called>
> #8  0x0000007f7cbbe5e0 in __lll_lock_wait () from /lib/aarch64-linux-gnu/libpthread.so.0
> #9  0x0000007f7cbb797c in pthread_mutex_lock () from /lib/aarch64-linux-gnu/libpthread.so.0
> #10 0x00000055886cb720 in mono_os_mutex_lock (mutex=0x55888227c8 <sgen_gc_mutex>) at ../../mono/utils/mono-os-mutex.h:103
> #11 mono_coop_mutex_lock (mutex=0x55888227c8 <sgen_gc_mutex>) at ../../mono/utils/mono-coop-mutex.h:54
> #12 sgen_gc_lock () at sgen-gc.c:3766
> #13 0x00000055886bcc6c in sgen_alloc_obj (vtable=0x55c4a088c8, size=40) at sgen-alloc.c:423
> #14 0x00000055886a2460 in mono_gc_alloc_obj (vtable=<optimized out>, size=<optimized out>) at sgen-mono.c:917
> #15 0x0000007f7c9c148c in ?? ()
> #16 0x0000007f5b5f9950 in ?? ()
> Backtrace stopped: previous frame inner to this frame (corrupt stack?)
> 
> Thread 14 (Thread 0x7f5b7fb1d0 (LWP 8860)):
> #0  0x0000007f7cbbdb1c in do_futex_wait () from /lib/aarch64-linux-gnu/libpthread.so.0
> #1  0x0000007f7cbbdbe8 in __new_sem_wait_slow () from /lib/aarch64-linux-gnu/libpthread.so.0
> #2  0x0000007f7cbbdd04 in sem_timedwait () from /lib/aarch64-linux-gnu/libpthread.so.0
> #3  0x0000005588651f08 in mono_os_sem_timedwait (flags=MONO_SEM_FLAGS_NONE, timeout_ms=1000, sem=0x7ef6326008) at ../../mono/utils/mono-os-semaphore.h:252
> #4  summarizer_state_wait (thread=<optimized out>) at threads.c:6522
> #5  mono_threads_summarize_execute (ctx=0x55888218e0 <suspend_signal_mask>, ctx@entry=0x7f5b7f84e0, out=0x7f440008c0, out@entry=0x7f5b7f84e8, hashes=0x7f5b7f84d0, hashes@entry=0x7f5b7f84f0, silent=silent@entry=0, working_mem=0x558871234c <suspend_signal_handler+228> "`\242C\371 \002", working_mem@entry=0x0, provided_size=546995930240, provided_size@entry=0) at threads.c:6584
> #6  0x00000055884df650 in sigterm_signal_handler (_dummy=15, _info=0x7f5b7f87f0, context=0x7f5b7f8870) at mini-posix.c:251
> #7  <signal handler called>
> #8  0x0000007f7cbbe5b0 in __lll_lock_wait () from /lib/aarch64-linux-gnu/libpthread.so.0
> #9  0x0000007f7cbb79e0 in pthread_mutex_lock () from /lib/aarch64-linux-gnu/libpthread.so.0
> #10 0x000000558859c978 in mono_os_mutex_lock (mutex=0x55c492bee0) at ../../mono/utils/mono-os-mutex.h:103
> #11 mono_coop_mutex_lock (mutex=0x55c492bee0) at ../../mono/utils/mono-coop-mutex.h:54
> #12 mono_domain_lock (domain=0x55c492bee0) at domain.c:1986
> #13 0x00000055884e43f0 in mono_codegen (cfg=cfg@entry=0x7f44009d60) at mini.c:2235
> #14 0x00000055884e6394 in mini_method_compile (method=method@entry=0x7f44014020, opts=opts@entry=374417919, domain=domain@entry=0x55c492bee0, flags=flags@entry=JIT_FLAG_RUN_CCTORS, parts=parts@entry=0, aot_method_index=aot_method_index@entry=-1) at mini.c:3847
> #15 0x00000055884e6e08 in mono_jit_compile_method_inner (method=method@entry=0x7f44014020, target_domain=target_domain@entry=0x55c492bee0, opt=374417919, opt@entry=1140856080, error=error@entry=0x7f5b7fa050) at mini.c:4036
> #16 0x000000558847d700 in mono_jit_compile_method_with_opt (method=0x7f44014020, opt=1140856080, jit_only=127, error=0x7f5b7fa050) at mini-runtime.c:2451
> #17 0x0000005588480168 in create_delegate_method_ptr (error=0x7f5b7fa050, method=0x7f44014020) at mini-runtime.c:3535
> #18 mini_init_delegate (delegate=..., error=0x7f5b7fa050) at mini-runtime.c:3560
> #19 0x000000558862c53c in mono_delegate_ctor_with_method (this_obj=this_obj@entry=..., target=target@entry=..., addr=addr@entry=0x0, method=<optimized out>, method@entry=0x7f44014020, error=error@entry=0x7f5b7fa050) at object.c:8601
> #20 0x00000055885d4574 in ves_icall_System_Delegate_CreateDelegate_internal (ref_type=..., ref_type@entry=..., target=target@entry=..., info=..., info@entry=..., throwOnBindFailure=throwOnBindFailure@entry=1 '\001', error=error@entry=0x7f5b7fa050) at icall.c:6854
> #21 0x00000055885db0c4 in ves_icall_System_Delegate_CreateDelegate_internal_raw (a0=0x7f5b7f9fc0, a1=0x7f5b7f9fc8, a2=0x7f5b7f9fd0, a3=1 '\001', error=0x7f5b7fa050) at ../../mono/metadata/icall-def.h:264
> #22 0x0000007f7806d054 in ?? ()
> #23 0x0000007f7c4ed4f8 in ?? ()
> Backtrace stopped: previous frame identical to this frame (corrupt stack?)
> 
> Thread 13 (Thread 0x7f5b9fc1d0 (LWP 8859)):
> #0  0x0000007f7cbbdb1c in do_futex_wait () from /lib/aarch64-linux-gnu/libpthread.so.0
> #1  0x0000007f7cbbdbe8 in __new_sem_wait_slow () from /lib/aarch64-linux-gnu/libpthread.so.0
> #2  0x0000007f7cbbdd04 in sem_timedwait () from /lib/aarch64-linux-gnu/libpthread.so.0
> #3  0x0000005588651f08 in mono_os_sem_timedwait (flags=MONO_SEM_FLAGS_NONE, timeout_ms=1000, sem=0x7ef6202008) at ../../mono/utils/mono-os-semaphore.h:252
> #4  summarizer_state_wait (thread=<optimized out>) at threads.c:6522
> #5  mono_threads_summarize_execute (ctx=0x55888218e0 <suspend_signal_mask>, ctx@entry=0x7f5b9f9a30, out=0x7f400008c0, out@entry=0x7f5b9f9a38, hashes=0x7f5b9f9a20, hashes@entry=0x7f5b9f9a40, silent=silent@entry=0, working_mem=0x558871234c <suspend_signal_handler+228> "`\242C\371 \002", working_mem@entry=0x0, provided_size=546998032848, provided_size@entry=0) at threads.c:6584
> #6  0x00000055884df650 in sigterm_signal_handler (_dummy=15, _info=0x7f5b9f9d40, context=0x7f5b9f9dc0) at mini-posix.c:251
> #7  <signal handler called>
> #8  0x0000007f7cbbe5e0 in __lll_lock_wait () from /lib/aarch64-linux-gnu/libpthread.so.0
> #9  0x0000007f7cbb797c in pthread_mutex_lock () from /lib/aarch64-linux-gnu/libpthread.so.0
> #10 0x00000055886cb720 in mono_os_mutex_lock (mutex=0x55888227c8 <sgen_gc_mutex>) at ../../mono/utils/mono-os-mutex.h:103
> #11 mono_coop_mutex_lock (mutex=0x55888227c8 <sgen_gc_mutex>) at ../../mono/utils/mono-coop-mutex.h:54
> #12 sgen_gc_lock () at sgen-gc.c:3766
> #13 0x00000055886a2554 in mono_gc_alloc_vector (vtable=0x7f000499f0, size=40, max_length=1) at sgen-mono.c:1316
> #14 0x0000007f7c9c3ac4 in ?? ()
> #15 0x0000007f40002df0 in ?? ()
> Backtrace stopped: previous frame inner to this frame (corrupt stack?)
> 
> Thread 12 (Thread 0x7f5bbfd1d0 (LWP 8858)):
> #0  0x0000007f7cbbdb1c in do_futex_wait () from /lib/aarch64-linux-gnu/libpthread.so.0
> #1  0x0000007f7cbbdbe8 in __new_sem_wait_slow () from /lib/aarch64-linux-gnu/libpthread.so.0
> #2  0x0000007f7cbbdd04 in sem_timedwait () from /lib/aarch64-linux-gnu/libpthread.so.0
> #3  0x0000005588651f08 in mono_os_sem_timedwait (flags=MONO_SEM_FLAGS_NONE, timeout_ms=1000, sem=0x7ef6127008) at ../../mono/utils/mono-os-semaphore.h:252
> #4  summarizer_state_wait (thread=<optimized out>) at threads.c:6522
> #5  mono_threads_summarize_execute (ctx=0x55888218e0 <suspend_signal_mask>, ctx@entry=0x7f5bbfa760, out=0x7f4c0008c0, out@entry=0x7f5bbfa768, hashes=0x7f5bbfa750, hashes@entry=0x7f5bbfa770, silent=silent@entry=0, working_mem=0x558871234c <suspend_signal_handler+228> "`\242C\371 \002", working_mem@entry=0x0, provided_size=547000133376, provided_size@entry=0) at threads.c:6584
> #6  0x00000055884df650 in sigterm_signal_handler (_dummy=15, _info=0x7f5bbfaa70, context=0x7f5bbfaaf0) at mini-posix.c:251
> #7  <signal handler called>
> #8  0x0000007f7cbbe5e0 in __lll_lock_wait () from /lib/aarch64-linux-gnu/libpthread.so.0
> #9  0x0000007f7cbb797c in pthread_mutex_lock () from /lib/aarch64-linux-gnu/libpthread.so.0
> #10 0x00000055886cb720 in mono_os_mutex_lock (mutex=0x55888227c8 <sgen_gc_mutex>) at ../../mono/utils/mono-os-mutex.h:103
> #11 mono_coop_mutex_lock (mutex=0x55888227c8 <sgen_gc_mutex>) at ../../mono/utils/mono-coop-mutex.h:54
> #12 sgen_gc_lock () at sgen-gc.c:3766
> #13 0x00000055886a2554 in mono_gc_alloc_vector (vtable=0x55c4a08fe0, size=40, max_length=1) at sgen-mono.c:1316
> #14 0x0000007f7c9c3ac4 in ?? ()
> #15 0x0000007f4c002df0 in ?? ()
> Backtrace stopped: previous frame inner to this frame (corrupt stack?)
> 
> Thread 11 (Thread 0x7f5bdfe1d0 (LWP 8857)):
> #0  0x0000007f7cbbdb1c in do_futex_wait () from /lib/aarch64-linux-gnu/libpthread.so.0
> #1  0x0000007f7cbbdbe8 in __new_sem_wait_slow () from /lib/aarch64-linux-gnu/libpthread.so.0
> #2  0x0000007f7cbbdd04 in sem_timedwait () from /lib/aarch64-linux-gnu/libpthread.so.0
> #3  0x0000005588651f08 in mono_os_sem_timedwait (flags=MONO_SEM_FLAGS_NONE, timeout_ms=1000, sem=0x7ef6003008) at ../../mono/utils/mono-os-semaphore.h:252
> #4  summarizer_state_wait (thread=<optimized out>) at threads.c:6522
> #5  mono_threads_summarize_execute (ctx=0x55888218e0 <suspend_signal_mask>, ctx@entry=0x7f5bdfbba0, out=0x7f480008c0, out@entry=0x7f5bdfbba8, hashes=0x7f5bdfbb90, hashes@entry=0x7f5bdfbbb0, silent=silent@entry=0, working_mem=0x558871234c <suspend_signal_handler+228> "`\242C\371 \002", working_mem@entry=0x0, provided_size=547002235712, provided_size@entry=0) at threads.c:6584
> #6  0x00000055884df650 in sigterm_signal_handler (_dummy=15, _info=0x7f5bdfbeb0, context=0x7f5bdfbf30) at mini-posix.c:251
> #7  <signal handler called>
> #8  0x0000007f7cbbe5e0 in __lll_lock_wait () from /lib/aarch64-linux-gnu/libpthread.so.0
> #9  0x0000007f7cbb797c in pthread_mutex_lock () from /lib/aarch64-linux-gnu/libpthread.so.0
> #10 0x00000055886cb720 in mono_os_mutex_lock (mutex=0x55888227c8 <sgen_gc_mutex>) at ../../mono/utils/mono-os-mutex.h:103
> #11 mono_coop_mutex_lock (mutex=0x55888227c8 <sgen_gc_mutex>) at ../../mono/utils/mono-coop-mutex.h:54
> #12 sgen_gc_lock () at sgen-gc.c:3766
> #13 0x00000055886a2554 in mono_gc_alloc_vector (vtable=0x55c49def28, size=96, max_length=8) at sgen-mono.c:1316
> #14 0x0000007f7c9c3ac4 in ?? ()
> #15 0x0000007f48002df0 in ?? ()
> Backtrace stopped: previous frame inner to this frame (corrupt stack?)
> 
> Thread 10 (Thread 0x7f5bfff1d0 (LWP 8856)):
> #0  0x0000007f7cbbdb1c in do_futex_wait () from /lib/aarch64-linux-gnu/libpthread.so.0
> #1  0x0000007f7cbbdbe8 in __new_sem_wait_slow () from /lib/aarch64-linux-gnu/libpthread.so.0
> #2  0x0000007f7cbbdd04 in sem_timedwait () from /lib/aarch64-linux-gnu/libpthread.so.0
> #3  0x0000005588651f08 in mono_os_sem_timedwait (flags=MONO_SEM_FLAGS_NONE, timeout_ms=1000, sem=0x7ef65b7008) at ../../mono/utils/mono-os-semaphore.h:252
> #4  summarizer_state_wait (thread=<optimized out>) at threads.c:6522
> #5  mono_threads_summarize_execute (ctx=0x55888218e0 <suspend_signal_mask>, ctx@entry=0x7f5bffc950, out=0x7f500008c0, out@entry=0x7f5bffc958, hashes=0x7f5bffc940, hashes@entry=0x7f5bffc960, silent=silent@entry=0, working_mem=0x558871234c <suspend_signal_handler+228> "`\242C\371 \002", working_mem@entry=0x0, provided_size=547004336368, provided_size@entry=0) at threads.c:6584
> #6  0x00000055884df650 in sigterm_signal_handler (_dummy=15, _info=0x7f5bffcc60, context=0x7f5bffcce0) at mini-posix.c:251
> #7  <signal handler called>
> #8  0x0000007f7cbbe5dc in __lll_lock_wait () from /lib/aarch64-linux-gnu/libpthread.so.0
> #9  0x0000007f7cbb797c in pthread_mutex_lock () from /lib/aarch64-linux-gnu/libpthread.so.0
> #10 0x00000055886cb720 in mono_os_mutex_lock (mutex=0x55888227c8 <sgen_gc_mutex>) at ../../mono/utils/mono-os-mutex.h:103
> #11 mono_coop_mutex_lock (mutex=0x55888227c8 <sgen_gc_mutex>) at ../../mono/utils/mono-coop-mutex.h:54
> #12 sgen_gc_lock () at sgen-gc.c:3766
> #13 0x00000055886bcc6c in sgen_alloc_obj (vtable=0x7f1c011fa0, size=72) at sgen-alloc.c:423
> #14 0x00000055886a2460 in mono_gc_alloc_obj (vtable=<optimized out>, size=<optimized out>) at sgen-mono.c:917
> #15 0x0000007f7c9c148c in ?? ()
> #16 0x0000007f7c9bec28 in ?? ()
> Backtrace stopped: previous frame inner to this frame (corrupt stack?)
> 
> Thread 9 (Thread 0x7f785821d0 (LWP 8854)):
> #0  0x0000007f7cbbdb1c in do_futex_wait () from /lib/aarch64-linux-gnu/libpthread.so.0
> #1  0x0000007f7cbbdbe8 in __new_sem_wait_slow () from /lib/aarch64-linux-gnu/libpthread.so.0
> #2  0x0000007f7cbbdd04 in sem_timedwait () from /lib/aarch64-linux-gnu/libpthread.so.0
> #3  0x0000005588651f08 in mono_os_sem_timedwait (flags=MONO_SEM_FLAGS_NONE, timeout_ms=1000, sem=0x7ef624b008) at ../../mono/utils/mono-os-semaphore.h:252
> #4  summarizer_state_wait (thread=<optimized out>) at threads.c:6522
> #5  mono_threads_summarize_execute (ctx=0x55888218e0 <suspend_signal_mask>, ctx@entry=0x7f7857fbb0, out=0x7f600008c0, out@entry=0x7f7857fbb8, hashes=0x7f7857fba0, hashes@entry=0x7f7857fbc0, silent=silent@entry=0, working_mem=0x558871234c <suspend_signal_handler+228> "`\242C\371 \002", working_mem@entry=0x0, provided_size=547479878480, provided_size@entry=0) at threads.c:6584
> #6  0x00000055884df650 in sigterm_signal_handler (_dummy=15, _info=0x7f7857fec0, context=0x7f7857ff40) at mini-posix.c:251
> #7  <signal handler called>
> #8  0x0000007f7cbbe5e0 in __lll_lock_wait () from /lib/aarch64-linux-gnu/libpthread.so.0
> #9  0x0000007f7cbb797c in pthread_mutex_lock () from /lib/aarch64-linux-gnu/libpthread.so.0
> #10 0x00000055886cb720 in mono_os_mutex_lock (mutex=0x55888227c8 <sgen_gc_mutex>) at ../../mono/utils/mono-os-mutex.h:103
> #11 mono_coop_mutex_lock (mutex=0x55888227c8 <sgen_gc_mutex>) at ../../mono/utils/mono-coop-mutex.h:54
> #12 sgen_gc_lock () at sgen-gc.c:3766
> #13 0x00000055886bcc6c in sgen_alloc_obj (vtable=0x55c4a088c8, size=40) at sgen-alloc.c:423
> #14 0x00000055886a2460 in mono_gc_alloc_obj (vtable=<optimized out>, size=<optimized out>) at sgen-mono.c:917
> #15 0x0000007f7c9c148c in ?? ()
> #16 0x0000007f78581950 in ?? ()
> Backtrace stopped: previous frame inner to this frame (corrupt stack?)
> 
> Thread 8 (Thread 0x7f787831d0 (LWP 8853)):
> #0  0x0000007f7cbbdb1c in do_futex_wait () from /lib/aarch64-linux-gnu/libpthread.so.0
> #1  0x0000007f7cbbdbe8 in __new_sem_wait_slow () from /lib/aarch64-linux-gnu/libpthread.so.0
> #2  0x0000007f7cbbdd04 in sem_timedwait () from /lib/aarch64-linux-gnu/libpthread.so.0
> #3  0x0000005588651f08 in mono_os_sem_timedwait (flags=MONO_SEM_FLAGS_NONE, timeout_ms=1000, sem=0x7ef6401008) at ../../mono/utils/mono-os-semaphore.h:252
> #4  summarizer_state_wait (thread=<optimized out>) at threads.c:6522
> #5  mono_threads_summarize_execute (ctx=0x55888218e0 <suspend_signal_mask>, ctx@entry=0x7f78780ba0, out=0x7f5c0008c0, out@entry=0x7f78780ba8, hashes=0x7f78780b90, hashes@entry=0x7f78780bb0, silent=silent@entry=0, working_mem=0x558871234c <suspend_signal_handler+228> "`\242C\371 \002", working_mem@entry=0x0, provided_size=547481979712, provided_size@entry=0) at threads.c:6584
> #6  0x00000055884df650 in sigterm_signal_handler (_dummy=15, _info=0x7f78780eb0, context=0x7f78780f30) at mini-posix.c:251
> #7  <signal handler called>
> #8  0x0000007f7cbbe5e0 in __lll_lock_wait () from /lib/aarch64-linux-gnu/libpthread.so.0
> #9  0x0000007f7cbb797c in pthread_mutex_lock () from /lib/aarch64-linux-gnu/libpthread.so.0
> #10 0x00000055886cb720 in mono_os_mutex_lock (mutex=0x55888227c8 <sgen_gc_mutex>) at ../../mono/utils/mono-os-mutex.h:103
> #11 mono_coop_mutex_lock (mutex=0x55888227c8 <sgen_gc_mutex>) at ../../mono/utils/mono-coop-mutex.h:54
> #12 sgen_gc_lock () at sgen-gc.c:3766
> #13 0x00000055886a2554 in mono_gc_alloc_vector (vtable=0x55c49def28, size=96, max_length=8) at sgen-mono.c:1316
> #14 0x0000007f7c9c3ac4 in ?? ()
> #15 0x0000007f5c002df0 in ?? ()
> Backtrace stopped: previous frame inner to this frame (corrupt stack?)
> 
> Thread 7 (Thread 0x7f789841d0 (LWP 8852)):
> #0  0x0000007f7cbbdb1c in do_futex_wait () from /lib/aarch64-linux-gnu/libpthread.so.0
> #1  0x0000007f7cbbdbe8 in __new_sem_wait_slow () from /lib/aarch64-linux-gnu/libpthread.so.0
> #2  0x0000007f7cbbdd04 in sem_timedwait () from /lib/aarch64-linux-gnu/libpthread.so.0
> #3  0x0000005588651f08 in mono_os_sem_timedwait (flags=MONO_SEM_FLAGS_NONE, timeout_ms=1000, sem=0x7ef61b9008) at ../../mono/utils/mono-os-semaphore.h:252
> #4  summarizer_state_wait (thread=<optimized out>) at threads.c:6522
> #5  mono_threads_summarize_execute (ctx=0x55888218e0 <suspend_signal_mask>, ctx@entry=0x7f78981ba0, out=0x7f680008c0, out@entry=0x7f78981ba8, hashes=0x7f78981b90, hashes@entry=0x7f78981bb0, silent=silent@entry=0, working_mem=0x558871234c <suspend_signal_handler+228> "`\242C\371 \002", working_mem@entry=0x0, provided_size=547484080960, provided_size@entry=0) at threads.c:6584
> #6  0x00000055884df650 in sigterm_signal_handler (_dummy=15, _info=0x7f78981eb0, context=0x7f78981f30) at mini-posix.c:251
> #7  <signal handler called>
> #8  0x0000007f7cbbe5e0 in __lll_lock_wait () from /lib/aarch64-linux-gnu/libpthread.so.0
> #9  0x0000007f7cbb797c in pthread_mutex_lock () from /lib/aarch64-linux-gnu/libpthread.so.0
> #10 0x00000055886cb720 in mono_os_mutex_lock (mutex=0x55888227c8 <sgen_gc_mutex>) at ../../mono/utils/mono-os-mutex.h:103
> #11 mono_coop_mutex_lock (mutex=0x55888227c8 <sgen_gc_mutex>) at ../../mono/utils/mono-coop-mutex.h:54
> #12 sgen_gc_lock () at sgen-gc.c:3766
> #13 0x00000055886a2554 in mono_gc_alloc_vector (vtable=0x55c49def28, size=96, max_length=8) at sgen-mono.c:1316
> #14 0x0000007f7c9c3ac4 in ?? ()
> #15 0x0000007f68002df0 in ?? ()
> Backtrace stopped: previous frame inner to this frame (corrupt stack?)
> 
> Thread 6 (Thread 0x7f78b851d0 (LWP 8851)):
> #0  0x0000007f7cbbdb1c in do_futex_wait () from /lib/aarch64-linux-gnu/libpthread.so.0
> #1  0x0000007f7cbbdbe8 in __new_sem_wait_slow () from /lib/aarch64-linux-gnu/libpthread.so.0
> #2  0x0000007f7cbbdd04 in sem_timedwait () from /lib/aarch64-linux-gnu/libpthread.so.0
> #3  0x0000005588651f08 in mono_os_sem_timedwait (flags=MONO_SEM_FLAGS_NONE, timeout_ms=1000, sem=0x7ef6170008) at ../../mono/utils/mono-os-semaphore.h:252
> #4  summarizer_state_wait (thread=<optimized out>) at threads.c:6522
> #5  mono_threads_summarize_execute (ctx=0x55888218e0 <suspend_signal_mask>, ctx@entry=0x7f78b82bb0, out=0x7f640008c0, out@entry=0x7f78b82bb8, hashes=0x7f78b82ba0, hashes@entry=0x7f78b82bc0, silent=silent@entry=0, working_mem=0x558871234c <suspend_signal_handler+228> "`\242C\371 \002", working_mem@entry=0x0, provided_size=547486182224, provided_size@entry=0) at threads.c:6584
> #6  0x00000055884df650 in sigterm_signal_handler (_dummy=15, _info=0x7f78b82ec0, context=0x7f78b82f40) at mini-posix.c:251
> #7  <signal handler called>
> #8  0x0000007f7cbbe5e0 in __lll_lock_wait () from /lib/aarch64-linux-gnu/libpthread.so.0
> #9  0x0000007f7cbb797c in pthread_mutex_lock () from /lib/aarch64-linux-gnu/libpthread.so.0
> #10 0x00000055886cb720 in mono_os_mutex_lock (mutex=0x55888227c8 <sgen_gc_mutex>) at ../../mono/utils/mono-os-mutex.h:103
> #11 mono_coop_mutex_lock (mutex=0x55888227c8 <sgen_gc_mutex>) at ../../mono/utils/mono-coop-mutex.h:54
> #12 sgen_gc_lock () at sgen-gc.c:3766
> #13 0x00000055886bcc6c in sgen_alloc_obj (vtable=0x55c4a088c8, size=40) at sgen-alloc.c:423
> #14 0x00000055886a2460 in mono_gc_alloc_obj (vtable=<optimized out>, size=<optimized out>) at sgen-mono.c:917
> #15 0x0000007f7c9c148c in ?? ()
> #16 0x0000007f78b84950 in ?? ()
> Backtrace stopped: previous frame inner to this frame (corrupt stack?)
> 
> Thread 5 (Thread 0x7f78d861d0 (LWP 8850)):
> #0  0x0000007f7cbbdb1c in do_futex_wait () from /lib/aarch64-linux-gnu/libpthread.so.0
> #1  0x0000007f7cbbdbe8 in __new_sem_wait_slow () from /lib/aarch64-linux-gnu/libpthread.so.0
> #2  0x0000007f7cbbdd04 in sem_timedwait () from /lib/aarch64-linux-gnu/libpthread.so.0
> #3  0x0000005588651f08 in mono_os_sem_timedwait (flags=MONO_SEM_FLAGS_NONE, timeout_ms=1000, sem=0x7ef5fba008) at ../../mono/utils/mono-os-semaphore.h:252
> #4  summarizer_state_wait (thread=<optimized out>) at threads.c:6522
> #5  mono_threads_summarize_execute (ctx=0x55888218e0 <suspend_signal_mask>, ctx@entry=0x7f78d83ba0, out=0x7f700008c0, out@entry=0x7f78d83ba8, hashes=0x7f78d83b90, hashes@entry=0x7f78d83bb0, silent=silent@entry=0, working_mem=0x558871234c <suspend_signal_handler+228> "`\242C\371 \002", working_mem@entry=0x0, provided_size=547488283456, provided_size@entry=0) at threads.c:6584
> #6  0x00000055884df650 in sigterm_signal_handler (_dummy=15, _info=0x7f78d83eb0, context=0x7f78d83f30) at mini-posix.c:251
> #7  <signal handler called>
> #8  0x0000007f7cbbe5e0 in __lll_lock_wait () from /lib/aarch64-linux-gnu/libpthread.so.0
> #9  0x0000007f7cbb797c in pthread_mutex_lock () from /lib/aarch64-linux-gnu/libpthread.so.0
> #10 0x00000055886cb720 in mono_os_mutex_lock (mutex=0x55888227c8 <sgen_gc_mutex>) at ../../mono/utils/mono-os-mutex.h:103
> #11 mono_coop_mutex_lock (mutex=0x55888227c8 <sgen_gc_mutex>) at ../../mono/utils/mono-coop-mutex.h:54
> #12 sgen_gc_lock () at sgen-gc.c:3766
> #13 0x00000055886a2554 in mono_gc_alloc_vector (vtable=0x55c49def28, size=96, max_length=8) at sgen-mono.c:1316
> #14 0x0000007f7c9c3ac4 in ?? ()
> #15 0x0000007f70002df0 in ?? ()
> Backtrace stopped: previous frame inner to this frame (corrupt stack?)
> 
> Thread 4 (Thread 0x7f78f871d0 (LWP 8849)):
> #0  0x0000007f7cbbdb1c in do_futex_wait () from /lib/aarch64-linux-gnu/libpthread.so.0
> #1  0x0000007f7cbbdbe8 in __new_sem_wait_slow () from /lib/aarch64-linux-gnu/libpthread.so.0
> #2  0x0000007f7cbbdd04 in sem_timedwait () from /lib/aarch64-linux-gnu/libpthread.so.0
> #3  0x0000005588651f08 in mono_os_sem_timedwait (flags=MONO_SEM_FLAGS_NONE, timeout_ms=1000, sem=0x7ef63b8008) at ../../mono/utils/mono-os-semaphore.h:252
> #4  summarizer_state_wait (thread=<optimized out>) at threads.c:6522
> #5  mono_threads_summarize_execute (ctx=0x55888218e0 <suspend_signal_mask>, ctx@entry=0x7f78f84d40, out=0x7f6c0008c0, out@entry=0x7f78f84d48, hashes=0x7f78f84d30, hashes@entry=0x7f78f84d50, silent=silent@entry=0, working_mem=0x558871234c <suspend_signal_handler+228> "`\242C\371 \002", working_mem@entry=0x0, provided_size=547490385120, provided_size@entry=0) at threads.c:6584
> #6  0x00000055884df650 in sigterm_signal_handler (_dummy=15, _info=0x7f78f85050, context=0x7f78f850d0) at mini-posix.c:251
> #7  <signal handler called>
> #8  0x0000007f7cbbe5dc in __lll_lock_wait () from /lib/aarch64-linux-gnu/libpthread.so.0
> #9  0x0000007f7cbb797c in pthread_mutex_lock () from /lib/aarch64-linux-gnu/libpthread.so.0
> #10 0x00000055886cb720 in mono_os_mutex_lock (mutex=0x55888227c8 <sgen_gc_mutex>) at ../../mono/utils/mono-os-mutex.h:103
> #11 mono_coop_mutex_lock (mutex=0x55888227c8 <sgen_gc_mutex>) at ../../mono/utils/mono-coop-mutex.h:54
> #12 sgen_gc_lock () at sgen-gc.c:3766
> #13 0x00000055886bcc6c in sgen_alloc_obj (vtable=0x55c4a08400, size=48) at sgen-alloc.c:423
> #14 0x00000055886a2460 in mono_gc_alloc_obj (vtable=<optimized out>, size=<optimized out>) at sgen-mono.c:917
> #15 0x0000007f7c9c148c in ?? ()
> #16 0x0000007f78f86301 in ?? ()
> Backtrace stopped: previous frame inner to this frame (corrupt stack?)
> 
> Thread 3 (Thread 0x7f79eca1d0 (LWP 8848)):
> #0  0x0000007f7cbbd8e0 in do_futex_wait.constprop () from /lib/aarch64-linux-gnu/libpthread.so.0
> #1  0x0000007f7cbbd998 in __new_sem_wait_slow.constprop.0 () from /lib/aarch64-linux-gnu/libpthread.so.0
> #2  0x0000005588698838 in mono_os_sem_wait (flags=MONO_SEM_FLAGS_ALERTABLE, sem=0x5588812208 <finalizer_sem>) at ../../mono/utils/mono-os-semaphore.h:203
> #3  mono_coop_sem_wait (flags=MONO_SEM_FLAGS_ALERTABLE, sem=0x5588812208 <finalizer_sem>) at ../../mono/utils/mono-coop-semaphore.h:41
> #4  finalizer_thread (unused=<optimized out>) at gc.c:969
> #5  0x000000558865124c in start_wrapper_internal (stack_ptr=<optimized out>, start_info=0x0) at threads.c:1220
> #6  start_wrapper (data=0x55c494d010) at threads.c:1293
> #7  0x0000007f7cbb5090 in start_thread () from /lib/aarch64-linux-gnu/libpthread.so.0
> #8  0x0000007f7cb2c15c in ?? () from /lib/aarch64-linux-gnu/libc.so.6
> 
> Thread 2 (Thread 0x7f7c3ff1d0 (LWP 8847)):
> #0  0x0000007f7cbbb33c in pthread_cond_wait@@GLIBC_2.17 () from /lib/aarch64-linux-gnu/libpthread.so.0
> #1  0x00000055886f6464 in mono_os_cond_wait (mutex=0x5588820c18 <lock>, cond=0x5588820c68 <work_cond>) at ../../mono/utils/mono-os-mutex.h:177
> #2  get_work (job=<synthetic pointer>, do_idle=<synthetic pointer>, work_context=<synthetic pointer>, worker_index=0) at sgen-thread-pool.c:165
> #3  thread_func (data=<optimized out>) at sgen-thread-pool.c:196
> #4  0x0000007f7cbb5090 in start_thread () from /lib/aarch64-linux-gnu/libpthread.so.0
> #5  0x0000007f7cb2c15c in ?? () from /lib/aarch64-linux-gnu/libc.so.6
> 
> Thread 1 (Thread 0x7f7cd10a40 (LWP 8846)):
> #0  0x0000007f7cbbdb1c in do_futex_wait () from /lib/aarch64-linux-gnu/libpthread.so.0
> #1  0x0000007f7cbbdbe8 in __new_sem_wait_slow () from /lib/aarch64-linux-gnu/libpthread.so.0
> #2  0x0000007f7cbbdd04 in sem_timedwait () from /lib/aarch64-linux-gnu/libpthread.so.0
> #3  0x0000005588651f08 in mono_os_sem_timedwait (flags=MONO_SEM_FLAGS_NONE, timeout_ms=1000, sem=0x7ef5edf008) at ../../mono/utils/mono-os-semaphore.h:252
> #4  summarizer_state_wait (thread=<optimized out>) at threads.c:6522
> #5  mono_threads_summarize_execute (ctx=0x55888218e0 <suspend_signal_mask>, ctx@entry=0x7fdbdd9490, out=0x55c4927220, out@entry=0x7fdbdd9498, hashes=0x7fdbdd9480, hashes@entry=0x7fdbdd94a0, silent=silent@entry=0, working_mem=0x558871234c <suspend_signal_handler+228> "`\242C\371 \002", working_mem@entry=0x0, provided_size=549149578288, provided_size@entry=0) at threads.c:6584
> #6  0x00000055884df650 in sigterm_signal_handler (_dummy=15, _info=0x7fdbdd97a0, context=0x7fdbdd9820) at mini-posix.c:251
> #7  <signal handler called>
> #8  0x0000007f7cbbb340 in pthread_cond_wait@@GLIBC_2.17 () from /lib/aarch64-linux-gnu/libpthread.so.0
> #9  0x00000055886f985c in mono_os_cond_wait (mutex=0x5588820ea0 <signal_mutex>, cond=0x7fdbddab40) at mono-os-mutex.h:177
> #10 mono_os_event_wait_multiple (events=0x7fdbddab90, nevents=nevents@entry=51, waitall=waitall@entry=0, timeout=4294967295, alertable=1) at os-event-unix.c:190
> #11 0x000000558871078c in mono_thread_info_wait_multiple_handle (thread_handles=thread_handles@entry=0x7fdbddae68, nhandles=51, background_change_event=background_change_event@entry=0x5588811260 <background_change_event>, waitall=waitall@entry=0, timeout=timeout@entry=4294967295, alertable=alertable@entry=1) at mono-threads.c:2011
> #12 0x00000055886474f4 in wait_for_tids (wait=wait@entry=0x7fdbddae68, timeout=timeout@entry=4294967295, check_state_change=check_state_change@entry=1) at threads.c:3529
> #13 0x000000558864bc98 in mono_thread_manage () at threads.c:3724
> #14 0x0000005588489674 in mono_main (argc=<optimized out>, argv=<optimized out>) at driver.c:2616
> #15 0x000000558847a280 in mono_main_with_options (argv=<optimized out>, argc=<optimized out>) at main.c:50
> #16 main (argc=<optimized out>, argv=<optimized out>) at main.c:408
> 
> =================================================================
> 	Basic Fault Adddress Reporting
> =================================================================
> Memory around native instruction pointer (0x7f7ca949fc):0x7f7ca949ec  02 00 80 d2 03 01 80 d2 e8 10 80 d2 01 00 00 d4  ................
> 0x7f7ca949fc  f3 0b 40 f9 e0 03 04 2a fd 7b d2 a8 c0 03 5f d6  ..@....*.{...._.
> 0x7f7ca94a0c  1f 20 03 d5 81 08 00 f0 21 20 47 f9 42 d0 3b d5  . ......! G.B.;.
> 0x7f7ca94a1c  e0 03 00 4b 04 00 80 12 40 68 21 b8 ef ff ff 17  ...K....@h!.....
> 
> =================================================================
> 	Managed Stacktrace:
> =================================================================
> =================================================================
> Aborted
@lambdageek

This comment has been minimized.

Copy link
Member

@lambdageek lambdageek commented Apr 23, 2019

@lewurm could you take a look at this one

@lewurm lewurm self-assigned this Apr 23, 2019
lewurm added a commit to lewurm/mono that referenced this issue Aug 30, 2019
Consider the following example:

```csharp
static void CommonCallTarget () { }

static void TailA () {
    tailcall CommonCallTarget ();
}

static void TailB () {
    tailcall CommonCallTarget ();
}
```

Since `TailA` and `TailB` are tailcalling into `CommonCallTarget`, the resolution at patch-time is a bit tricky, that is, since it's a jump-like instruction the patching machinery won't know where it was called from. That's why we maintain a global hashtable `jump_target_hash` where each jump-site is signed up to be patched. At patch-time we know the target method (in the example `CommonCallTarget`), but since we don't know where we are coming from, we will just apply all patches for that target.

This works since ages, so why did it crash on arm64 sometimes?
When the patching happens, we check if the displacement between jump-site and target fits into it (24bit). If not, which happens not very often, we have to allocate a _thunk_:
https://github.com/mono/mono/blob/36296ce291f8a7b19de3eccb7a32c7e4ed9df8f2/mono/mini/mini-arm64.c#L928-L942

So instead of jumping to the target directly, we'll branch to the thunk. This is a little trampoline that loads the full address of the target and then finally branches to it. This one will live close-by the jump-site, because during compilation we will reserve specifically for that scenario some space after the generated code. For this, however, we need the JitInfo of the jump-site. And that's where the origin of the race is. Let's say:

* Thread A compiles TailA, and then jumps into it. Thus one patch point is in the jump_target_hash.
* Now Thread B compiles TailB, registers the patch point but has _not_ yet registered its JitInfo.
* Then Thread A continues, does the tailcall into CommonCallTarget, enters the patching machinery, which sees two patches. Now assume when applying the patch for TailB the displacement is too large, thus it tries to allocate a thunk but can't find the relevant JitInfo for it. So it crashes as reported in mono#14080

As far as I can tell this only affects ARM64, ARM and PPC.

Fixes mono#14080
lewurm added a commit to lewurm/mono that referenced this issue Aug 30, 2019
Consider the following example:

```csharp
static void CommonCallTarget () { }

static void TailA () {
    tailcall CommonCallTarget ();
}

static void TailB () {
    tailcall CommonCallTarget ();
}
```

Since `TailA` and `TailB` are tailcalling into `CommonCallTarget`, the resolution at patch-time is a bit tricky, that is, since it's a jump-like instruction the patching machinery won't know where it was called from. That's why we maintain a global hashtable `jump_target_hash` where each jump-site is signed up to be patched. At patch-time we know the target method (in the example `CommonCallTarget`), but since we don't know where we are coming from, we will just apply all patches for that target.

This works since ages, so why did it crash on arm64 sometimes?
When the patching happens, we check if the displacement between jump-site and target fits into it (24bit). If not, which happens not very often, we have to allocate a _thunk_:
https://github.com/mono/mono/blob/36296ce291f8a7b19de3eccb7a32c7e4ed9df8f2/mono/mini/mini-arm64.c#L928-L942

So instead of jumping to the target directly, we'll branch to the thunk. This is a little trampoline that loads the full address of the target and then finally branches to it. This one will live close-by the jump-site, because during compilation we will reserve specifically for that scenario some space after the generated code. For this, however, we need the JitInfo of the jump-site. And that's where the origin of the race is. Let's say:

* Thread A compiles TailA, and then jumps into it. Thus one patch point is in the jump_target_hash.
* Now Thread B compiles TailB, registers the patch point but has _not_ yet registered its JitInfo.
* Then Thread A continues, does the tailcall into CommonCallTarget, enters the patching machinery, which sees two patches. Now assume when applying the patch for TailB the displacement is too large, thus it tries to allocate a thunk but can't find the relevant JitInfo for it. So it crashes as reported in mono#14080

As far as I can tell this only affects ARM64, ARM and PPC.

Fixes mono#14080
lewurm added a commit to lewurm/mono that referenced this issue Aug 30, 2019
Fixes mono#14080

Consider the following example:

```csharp
static void CommonCallTarget () { }

static void TailA () {
    tailcall CommonCallTarget ();
}

static void TailB () {
    tailcall CommonCallTarget ();
}
```

Since `TailA` and `TailB` are tailcalling into `CommonCallTarget`, the resolution at patch-time is a bit tricky, that is, since it's a jump-like instruction the patching machinery won't know where it was called from. That's why we maintain a global hashtable `jump_target_hash` where each jump-site is signed up to be patched. At patch-time we know the target method (in the example `CommonCallTarget`), but since we don't know where we are coming from, we will just apply all patches for that target.

This works since ages, so why did it crash on arm64 sometimes?
When the patching happens, we check if the displacement between jump-site and target fits into it (24bit). If not, which happens not very often, we have to allocate a _thunk_:
https://github.com/mono/mono/blob/36296ce291f8a7b19de3eccb7a32c7e4ed9df8f2/mono/mini/mini-arm64.c#L928-L942

So instead of jumping to the target directly, we'll branch to the thunk. This is a little trampoline that loads the full address of the target and then finally branches to it. This one will live close-by the jump-site, because during compilation we will reserve specifically for that scenario some space after the generated code. For this, however, we need the JitInfo of the jump-site. And that's where the origin of the race is. Let's say:

* Thread A compiles `TailA`, and then jumps into it. Thus one patch point is in the `jump_target_hash`.
* Now Thread B compiles `TailB`, registers the patch point but has _not_ yet registered its JitInfo.
* Then Thread A continues, does the tailcall into `CommonCallTarget`, enters the patching machinery, which sees two patches. Now assume when applying the patch for `TailB` the displacement is too large, thus it tries to allocate a thunk but can't find the relevant JitInfo for it that it needs to emit the thunk. So it crashes as reported in mono#14080

As far as I can tell this only affects ARM64, ARM and PPC.
@lewurm

This comment has been minimized.

Copy link
Member

@lewurm lewurm commented Aug 30, 2019

Thank you @kdg3737, the reproducer from #14080 (comment) was very helpful!

Sorry that it took so long, here is the fix: #16589

monojenkins added a commit that referenced this issue Aug 30, 2019
[mini] publish global patches after JitInfo has been added

Fixes #14080

Consider the following example:

```csharp
static void CommonCallTarget () { }

static void TailA () {
    tailcall CommonCallTarget ();
}

static void TailB () {
    tailcall CommonCallTarget ();
}
```

Since `TailA` and `TailB` are tailcalling into `CommonCallTarget`, the resolution at patch-time is a bit tricky, that is, since it's a jump-like instruction the patching machinery won't know where it was called from. That's why we maintain a global hashtable `jump_target_hash` where each jump-site is signed up to be patched. At patch-time we know the target method (in the example `CommonCallTarget`), but since we don't know where we are coming from, we will just apply all patches for that target.

This works since ages, so why did it crash on arm64 sometimes?
When the patching happens, we check if the displacement between jump-site and target fits into it (24bit). If not, which happens not very often, we have to allocate a _thunk_:
https://github.com/mono/mono/blob/36296ce291f8a7b19de3eccb7a32c7e4ed9df8f2/mono/mini/mini-arm64.c#L928-L942

So instead of jumping to the target directly, we'll branch to the thunk. This is a little trampoline that loads the full address of the target and then finally branches to it. This one will live close-by the jump-site, because during compilation we will reserve specifically for that scenario some space after the generated code. For this, however, we need the JitInfo of the jump-site. And that's where the origin of the race is. Let's say:

* Thread A compiles `TailA`, and then jumps into it. Thus one patch point is in the `jump_target_hash`.
* Now Thread B compiles `TailB`, registers the patch point but has _not_ yet registered its JitInfo.
* Then Thread A continues, does the tailcall into `CommonCallTarget`, enters the patching machinery, which sees two patches. Now assume when applying the patch for `TailB` the displacement is too large, thus it tries to allocate a thunk but can't find the relevant JitInfo for it that it needs to emit the thunk. So it crashes as reported in #14080

As far as I can tell this only affects ARM64, ARM and PPC.



<!--
Thank you for your Pull Request!

If you are new to contributing to Mono, please try to do your best at conforming to our coding guidelines http://www.mono-project.com/community/contributing/coding-guidelines/ but don't worry if you get something wrong. One of the project members will help you to get things landed.

Does your pull request fix any of the existing issues? Please use the following format: Fixes #issue-number
-->
monojenkins added a commit to monojenkins/mono that referenced this issue Sep 9, 2019
Fixes mono#14080

Consider the following example:

```csharp
static void CommonCallTarget () { }

static void TailA () {
    tailcall CommonCallTarget ();
}

static void TailB () {
    tailcall CommonCallTarget ();
}
```

Since `TailA` and `TailB` are tailcalling into `CommonCallTarget`, the resolution at patch-time is a bit tricky, that is, since it's a jump-like instruction the patching machinery won't know where it was called from. That's why we maintain a global hashtable `jump_target_hash` where each jump-site is signed up to be patched. At patch-time we know the target method (in the example `CommonCallTarget`), but since we don't know where we are coming from, we will just apply all patches for that target.

This works since ages, so why did it crash on arm64 sometimes?
When the patching happens, we check if the displacement between jump-site and target fits into it (24bit). If not, which happens not very often, we have to allocate a _thunk_:
https://github.com/mono/mono/blob/36296ce291f8a7b19de3eccb7a32c7e4ed9df8f2/mono/mini/mini-arm64.c#L928-L942

So instead of jumping to the target directly, we'll branch to the thunk. This is a little trampoline that loads the full address of the target and then finally branches to it. This one will live close-by the jump-site, because during compilation we will reserve specifically for that scenario some space after the generated code. For this, however, we need the JitInfo of the jump-site. And that's where the origin of the race is. Let's say:

* Thread A compiles `TailA`, and then jumps into it. Thus one patch point is in the `jump_target_hash`.
* Now Thread B compiles `TailB`, registers the patch point but has _not_ yet registered its JitInfo.
* Then Thread A continues, does the tailcall into `CommonCallTarget`, enters the patching machinery, which sees two patches. Now assume when applying the patch for `TailB` the displacement is too large, thus it tries to allocate a thunk but can't find the relevant JitInfo for it that it needs to emit the thunk. So it crashes as reported in mono#14080

As far as I can tell this only affects ARM64, ARM and PPC.
marek-safar added a commit that referenced this issue Sep 9, 2019
Fixes #14080

Consider the following example:

```csharp
static void CommonCallTarget () { }

static void TailA () {
    tailcall CommonCallTarget ();
}

static void TailB () {
    tailcall CommonCallTarget ();
}
```

Since `TailA` and `TailB` are tailcalling into `CommonCallTarget`, the resolution at patch-time is a bit tricky, that is, since it's a jump-like instruction the patching machinery won't know where it was called from. That's why we maintain a global hashtable `jump_target_hash` where each jump-site is signed up to be patched. At patch-time we know the target method (in the example `CommonCallTarget`), but since we don't know where we are coming from, we will just apply all patches for that target.

This works since ages, so why did it crash on arm64 sometimes?
When the patching happens, we check if the displacement between jump-site and target fits into it (24bit). If not, which happens not very often, we have to allocate a _thunk_:
https://github.com/mono/mono/blob/36296ce291f8a7b19de3eccb7a32c7e4ed9df8f2/mono/mini/mini-arm64.c#L928-L942

So instead of jumping to the target directly, we'll branch to the thunk. This is a little trampoline that loads the full address of the target and then finally branches to it. This one will live close-by the jump-site, because during compilation we will reserve specifically for that scenario some space after the generated code. For this, however, we need the JitInfo of the jump-site. And that's where the origin of the race is. Let's say:

* Thread A compiles `TailA`, and then jumps into it. Thus one patch point is in the `jump_target_hash`.
* Now Thread B compiles `TailB`, registers the patch point but has _not_ yet registered its JitInfo.
* Then Thread A continues, does the tailcall into `CommonCallTarget`, enters the patching machinery, which sees two patches. Now assume when applying the patch for `TailB` the displacement is too large, thus it tries to allocate a thunk but can't find the relevant JitInfo for it that it needs to emit the thunk. So it crashes as reported in #14080

As far as I can tell this only affects ARM64, ARM and PPC.
@kdg3737

This comment has been minimized.

Copy link
Author

@kdg3737 kdg3737 commented Sep 11, 2019

The 'condition ji not met' error seems to be solved in the mono 6.7 I just built, however, my test program/loop still randomly crashes with the following:

System.ArgumentNullException: Value cannot be null.
Parameter name: returnLabel
at System.Dynamic.DynamicMetaObjectBinder.Bind (System.Object[] args, System.Collections.ObjectModel.ReadOnlyCollection1[T] parameters, System.Linq.Expressions.LabelTarget returnLabel) [0x001cd] in /persistent/mono/external/corefx/src/System.Linq.Expressions/src/System/Dynamic/DynamicMetaObjectBinder.cs:141 at System.Runtime.CompilerServices.CallSiteBinder.BindCore[T] (System.Runtime.CompilerServices.CallSite1[T] site, System.Object[] args) [0x0001c] in /persistent/mono/external/corefx/src/System.Linq.Expressions/src/System/Runtime/CompilerServices/CallSiteBinder.cs:129
at System.Dynamic.UpdateDelegates.UpdateAndExecute2[T0,T1,TRet] (System.Runtime.CompilerServices.CallSite site, T0 arg0, T1 arg1) [0x00126] in /persistent/mono/external/corefx/src/System.Linq.Expressions/src/System/Dynamic/UpdateDelegates.Generated.cs:263
at (wrapper delegate-invoke) System.Func4[System.Runtime.CompilerServices.CallSite,System.Object,System.Double,System.Object].invoke_TResult_T1_T2_T3(System.Runtime.CompilerServices.CallSite,object,double) at InverterScan.Program+<>c.<Main>b__46_0 () [0x00014] in C:\projects\Iridium\InverterScan\Program.cs:1462 at System.Threading.ThreadHelper.ThreadStart_Context (System.Object state) [0x00017] in /persistent/mono/mcs/class/referencesource/mscorlib/system/threading/thread.cs:74 at System.Threading.ExecutionContext.RunInternal (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, System.Boolean preserveSyncCtx) [0x0008d] in /persistent/mono/mcs/class/referencesource/mscorlib/system/threading/executioncontext.cs:968 at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, System.Boolean preserveSyncCtx) [0x00000] in /persistent/mono/mcs/class/referencesource/mscorlib/system/threading/executioncontext.cs:910 at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state) [0x00031] in /persistent/mono/mcs/class/referencesource/mscorlib/system/threading/executioncontext.cs:899 at System.Threading.ThreadHelper.ThreadStart () [0x0000b] in /persistent/mono/mcs/class/referencesource/mscorlib/system/threading/thread.cs:111 [ERROR] FATAL UNHANDLED EXCEPTION: System.ArgumentNullException: Value cannot be null. Parameter name: returnLabel at System.Dynamic.DynamicMetaObjectBinder.Bind (System.Object[] args, System.Collections.ObjectModel.ReadOnlyCollection1[T] parameters, System.Linq.Expressions.LabelTarget returnLabel) [0x001cd] in /persistent/mono/external/corefx/src/System.Linq.Expressions/src/System/Dynamic/DynamicMetaObjectBinder.cs:141
at System.Runtime.CompilerServices.CallSiteBinder.BindCore[T] (System.Runtime.CompilerServices.CallSite1[T] site, System.Object[] args) [0x0001c] in /persistent/mono/external/corefx/src/System.Linq.Expressions/src/System/Runtime/CompilerServices/CallSiteBinder.cs:129 at System.Dynamic.UpdateDelegates.UpdateAndExecute2[T0,T1,TRet] (System.Runtime.CompilerServices.CallSite site, T0 arg0, T1 arg1) [0x00126] in /persistent/mono/external/corefx/src/System.Linq.Expressions/src/System/Dynamic/UpdateDelegates.Generated.cs:263 at (wrapper delegate-invoke) System.Func4[System.Runtime.CompilerServices.CallSite,System.Object,System.Double,System.Object].invoke_TResult_T1_T2_T3(System.Runtime.CompilerServices.CallSite,object,double)
at InverterScan.Program+<>c.

b__46_0 () [0x00014] in C:\projects\Iridium\InverterScan\Program.cs:1462
at System.Threading.ThreadHelper.ThreadStart_Context (System.Object state) [0x00017] in /persistent/mono/mcs/class/referencesource/mscorlib/system/threading/thread.cs:74
at System.Threading.ExecutionContext.RunInternal (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, System.Boolean preserveSyncCtx) [0x0008d] in /persistent/mono/mcs/class/referencesource/mscorlib/system/threading/executioncontext.cs:968
at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, System.Boolean preserveSyncCtx) [0x00000] in /persistent/mono/mcs/class/referencesource/mscorlib/system/threading/executioncontext.cs:910
at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state) [0x00031] in /persistent/mono/mcs/class/referencesource/mscorlib/system/threading/executioncontext.cs:899
at System.Threading.ThreadHelper.ThreadStart () [0x0000b] in /persistent/mono/mcs/class/referencesource/mscorlib/system/threading/thread.cs:111

@kdg3737

This comment has been minimized.

Copy link
Author

@kdg3737 kdg3737 commented Sep 11, 2019

Seems to be ARM specific as well, no errors on x64 ubuntu, mono 5.18.

@lewurm

This comment has been minimized.

Copy link
Member

@lewurm lewurm commented Sep 11, 2019

@kdg3737 😕 could you please open a new issue, ideally with a reproducer attached?

@kdg3737

This comment has been minimized.

Copy link
Author

@kdg3737 kdg3737 commented Sep 11, 2019

Ok, see #16770
I tracked it down to an Expression.Label call which seems to be returning null for some reason, CallSiteBinder.cs line 82. Good luck & thanks.

@lewurm

This comment has been minimized.

Copy link
Member

@lewurm lewurm commented Sep 11, 2019

@kdg3737 oh, it's the same repro?

I had it running after my fix, it completed the 10000 iterations. How long does it take to repro for you?

Will try again.

@kdg3737

This comment has been minimized.

Copy link
Author

@kdg3737 kdg3737 commented Sep 11, 2019

Couple of minutes, maybe 1% of the calls. I could give you access to my hardware (rock64 board) and show you, if that helps.

@lewurm

This comment has been minimized.

Copy link
Member

@lewurm lewurm commented Sep 11, 2019

@kdg3737 thanks for the offer. I'll try on my machine first, and if I can't repro I'll get back to you.

In the meanwhile, could you try adding --clr-memory-model when calling mono?

@kdg3737

This comment has been minimized.

Copy link
Author

@kdg3737 kdg3737 commented Sep 11, 2019

Running with 'mono --clr-memory-model YOUREXE.exe' gives me the same error.

@kdg3737

This comment has been minimized.

Copy link
Author

@kdg3737 kdg3737 commented Sep 11, 2019

Not sure if it's relevant but here's how I compiled and installed mono:

MONO_TLS_PROVIDER=btls ./autogen.sh --prefix=/usr/local --enable-nls=no --with-csc=mcs
MONO_TLS_PROVIDER=btls make get-monolite-latest
MONO_TLS_PROVIDER=btls make -j4
sudo make install

jonpryor added a commit to xamarin/xamarin-android that referenced this issue Dec 3, 2019
Changes: mono/api-snapshot@fc50bc4...45a61d9

        $ git diff --shortstat fc50bc4f...45a61d93
         22 files changed, 775 insertions(+), 474 deletions(-)

Changes: mono/cecil@a6c8f5e...a6a7f5c

        $ git diff --shortstat a6c8f5e1...a6a7f5c0
         55 files changed, 818 insertions(+), 530 deletions(-)

Changes: mono/corefx@1f87de3...49f1c45

        $ git diff --shortstat e4f7102b...49f1c453
         38 files changed, 1171 insertions(+), 419 deletions(-)

Changes: mono/linker@ebe2a1f...e8d054b

        $ git diff --shortstat ebe2a1f4...e8d054bf
         137 files changed, 5360 insertions(+), 1781 deletions(-)

Changes: mono/mono@8946e49...18920a8

        $ git diff --shortstat 8946e49a...18920a83
         1811 files changed, 47240 insertions(+), 48331 deletions(-)

Changes: xamarin/xamarin-android-api-compatibility@a61271e...50a3c52

        $ git diff --shortstat a61271e0...50a3c52d
         1 file changed, 2 insertions(+), 791 deletions(-)

Fixes: #3619

Context: https://dev.azure.com/devdiv/DevDiv/_workitems/edit/1005448
Context: https://devdiv.visualstudio.com/DefaultCollection/DevDiv/_workitems/edit/967582
Context: dotnet/coreclr#26370
Context: dotnet/coreclr#26479
Context: dotnet/corefx#40455
Context: dotnet/corefx#40578
Context: mono/mono#7377
Context: mono/mono#12421
Context: mono/mono#12586
Context: mono/mono#14080
Context: mono/mono#14725
Context: mono/mono#14772
Context: mono/mono#15261
Context: mono/mono#15262
Context: mono/mono#15263
Context: mono/mono#15307
Context: mono/mono#15308
Context: mono/mono#15310
Context: mono/mono#15646
Context: mono/mono#15687
Context: mono/mono#15805
Context: mono/mono#15992
Context: mono/mono#15994
Context: mono/mono#15999
Context: mono/mono#16032
Context: mono/mono#16034
Context: mono/mono#16046
Context: mono/mono#16192
Context: mono/mono#16308
Context: mono/mono#16310
Context: mono/mono#16369
Context: mono/mono#16380
Context: mono/mono#16381
Context: mono/mono#16395
Context: mono/mono#16411
Context: mono/mono#16415
Context: mono/mono#16486
Context: mono/mono#16570
Context: mono/mono#16605
Context: mono/mono#16616
Context: mono/mono#16689
Context: mono/mono#16701
Context: mono/mono#16712
Context: mono/mono#16742
Context: mono/mono#16759
Context: mono/mono#16803
Context: mono/mono#16808
Context: mono/mono#16824
Context: mono/mono#16876
Context: mono/mono#16879
Context: mono/mono#16918
Context: mono/mono#16943
Context: mono/mono#16950
Context: mono/mono#16974
Context: mono/mono#17004
Context: mono/mono#17017
Context: mono/mono#17038
Context: mono/mono#17040
Context: mono/mono#17083
Context: mono/mono#17084
Context: mono/mono#17133
Context: mono/mono#17139
Context: mono/mono#17151
Context: mono/mono#17180
Context: mono/mono#17278
Context: mono/mono#17549
Context: mono/mono#17569
Context: mono/mono#17665
Context: mono/mono#17687
Context: mono/mono#17737
Context: mono/mono#17790
Context: mono/mono#17924
Context: mono/mono#17931
Context: https://github.com/mono/mono/issues/26758
Context: https://github.com/mono/mono/issues/37913
Context: xamarin/xamarin-macios#7005
jonpryor added a commit to xamarin/xamarin-android that referenced this issue Dec 3, 2019
Changes: mono/api-snapshot@fc50bc4...45a61d9

        $ git diff --shortstat fc50bc4f...45a61d93
         22 files changed, 775 insertions(+), 474 deletions(-)

Changes: mono/cecil@a6c8f5e...a6a7f5c

        $ git diff --shortstat a6c8f5e1...a6a7f5c0
         55 files changed, 818 insertions(+), 530 deletions(-)

Changes: mono/corefx@1f87de3...49f1c45

        $ git diff --shortstat e4f7102b...49f1c453
         38 files changed, 1171 insertions(+), 419 deletions(-)

Changes: mono/linker@ebe2a1f...e8d054b

        $ git diff --shortstat ebe2a1f4...e8d054bf
         137 files changed, 5360 insertions(+), 1781 deletions(-)

Changes: mono/mono@8946e49...18920a8

        $ git diff --shortstat 8946e49a...18920a83
         1811 files changed, 47240 insertions(+), 48331 deletions(-)

Changes: xamarin/xamarin-android-api-compatibility@a61271e...50a3c52

        $ git diff --shortstat a61271e0...50a3c52d
         1 file changed, 2 insertions(+), 791 deletions(-)

Fixes: #3619

Context: https://dev.azure.com/devdiv/DevDiv/_workitems/edit/1005448
Context: https://devdiv.visualstudio.com/DefaultCollection/DevDiv/_workitems/edit/967582
Context: dotnet/coreclr#26370
Context: dotnet/coreclr#26479
Context: dotnet/corefx#40455
Context: dotnet/corefx#40578
Context: mono/mono#7377
Context: mono/mono#12421
Context: mono/mono#12586
Context: mono/mono#14080
Context: mono/mono#14725
Context: mono/mono#14772
Context: mono/mono#15261
Context: mono/mono#15262
Context: mono/mono#15263
Context: mono/mono#15307
Context: mono/mono#15308
Context: mono/mono#15310
Context: mono/mono#15646
Context: mono/mono#15687
Context: mono/mono#15805
Context: mono/mono#15992
Context: mono/mono#15994
Context: mono/mono#15999
Context: mono/mono#16032
Context: mono/mono#16034
Context: mono/mono#16046
Context: mono/mono#16192
Context: mono/mono#16308
Context: mono/mono#16310
Context: mono/mono#16369
Context: mono/mono#16380
Context: mono/mono#16381
Context: mono/mono#16395
Context: mono/mono#16411
Context: mono/mono#16415
Context: mono/mono#16486
Context: mono/mono#16570
Context: mono/mono#16605
Context: mono/mono#16616
Context: mono/mono#16689
Context: mono/mono#16701
Context: mono/mono#16712
Context: mono/mono#16742
Context: mono/mono#16759
Context: mono/mono#16803
Context: mono/mono#16808
Context: mono/mono#16824
Context: mono/mono#16876
Context: mono/mono#16879
Context: mono/mono#16918
Context: mono/mono#16943
Context: mono/mono#16950
Context: mono/mono#16974
Context: mono/mono#17004
Context: mono/mono#17017
Context: mono/mono#17038
Context: mono/mono#17040
Context: mono/mono#17083
Context: mono/mono#17084
Context: mono/mono#17133
Context: mono/mono#17139
Context: mono/mono#17151
Context: mono/mono#17180
Context: mono/mono#17278
Context: mono/mono#17549
Context: mono/mono#17569
Context: mono/mono#17665
Context: mono/mono#17687
Context: mono/mono#17737
Context: mono/mono#17790
Context: mono/mono#17924
Context: mono/mono#17931
Context: https://github.com/mono/mono/issues/26758
Context: https://github.com/mono/mono/issues/37913
Context: xamarin/xamarin-macios#7005
ManickaP pushed a commit to ManickaP/runtime that referenced this issue Jan 20, 2020
…#16589)

[mini] publish global patches after JitInfo has been added

Fixes mono/mono#14080

Consider the following example:

```csharp
static void CommonCallTarget () { }

static void TailA () {
    tailcall CommonCallTarget ();
}

static void TailB () {
    tailcall CommonCallTarget ();
}
```

Since `TailA` and `TailB` are tailcalling into `CommonCallTarget`, the resolution at patch-time is a bit tricky, that is, since it's a jump-like instruction the patching machinery won't know where it was called from. That's why we maintain a global hashtable `jump_target_hash` where each jump-site is signed up to be patched. At patch-time we know the target method (in the example `CommonCallTarget`), but since we don't know where we are coming from, we will just apply all patches for that target.

This works since ages, so why did it crash on arm64 sometimes?
When the patching happens, we check if the displacement between jump-site and target fits into it (24bit). If not, which happens not very often, we have to allocate a _thunk_:
https://github.com/mono/mono/blob/mono/mono@36296ce291f8a7b19de3eccb7a32c7e4ed9df8f2/mono/mini/mini-arm64.c#L928-L942

So instead of jumping to the target directly, we'll branch to the thunk. This is a little trampoline that loads the full address of the target and then finally branches to it. This one will live close-by the jump-site, because during compilation we will reserve specifically for that scenario some space after the generated code. For this, however, we need the JitInfo of the jump-site. And that's where the origin of the race is. Let's say:

* Thread A compiles `TailA`, and then jumps into it. Thus one patch point is in the `jump_target_hash`.
* Now Thread B compiles `TailB`, registers the patch point but has _not_ yet registered its JitInfo.
* Then Thread A continues, does the tailcall into `CommonCallTarget`, enters the patching machinery, which sees two patches. Now assume when applying the patch for `TailB` the displacement is too large, thus it tries to allocate a thunk but can't find the relevant JitInfo for it that it needs to emit the thunk. So it crashes as reported in mono/mono#14080

As far as I can tell this only affects ARM64, ARM and PPC.



<!--
Thank you for your Pull Request!

If you are new to contributing to Mono, please try to do your best at conforming to our coding guidelines http://www.mono-project.com/community/contributing/coding-guidelines/ but don't worry if you get something wrong. One of the project members will help you to get things landed.

Does your pull request fix any of the existing issues? Please use the following format: Fixes #issue-number
-->



Commit migrated from mono/mono@06e63b3
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
4 participants
You can’t perform that action at this time.