Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Assertion when the process runs out of memory #10415

Closed
benjaminknox opened this issue Aug 31, 2018 · 14 comments
Assignees
Projects

Comments

@benjaminknox
Copy link

@benjaminknox benjaminknox commented Aug 31, 2018

Steps to Reproduce

  1. Compiled Debug or Release with Visual Studio
  2. Moved EXE files to CentOS host
  3. Ran exe with mono

Current Behavior

Currently, the program runs for a while then crashes giving the above

Expected Behavior

The program doesn't crash and continues to run

On which platforms did you notice this

[ ] macOS
[ x ] Linux
[ ] Windows

Version Used:
Mono JIT compiler version 5.14.0.177 (tarball Sat Aug 4 08:32:47 UTC 2018)
Copyright (C) 2002-2014 Novell, Inc, Xamarin Inc and Contributors. www.mono-project.com
TLS: __thread
SIGSEGV: altstack
Notifications: epoll
Architecture: amd64
Disabled: none
Misc: softdebug
Interpreter: yes
LLVM: yes(3.6.0svn-mono-/)
GC: sgen (concurrent by default)

Stacktrace

Assertion at mini-exceptions.c:2621, condition `gaddr == tls->stack_ovf_guard_base' not met

Stacktrace:

/proc/self/maps:
00400000-00824000 r-xp 00000000 08:02 73147316                           /usr/bin/mono-sgen
00a24000-00a2b000 r--p 00424000 08:02 73147316                           /usr/bin/mono-sgen
00a2b000-00a30000 rw-p 0042b000 08:02 73147316                           /usr/bin/mono-sgen
00a30000-00a46000 rw-p 00000000 00:00 0 
01245000-0191d000 rw-p 00000000 00:00 0                                  [heap]
4147c000-4148c000 rwxp 00000000 00:00 0 
419d5000-41b15000 rwxp 00000000 00:00 0 
7fa8f6100000-7fa8f6600000 rw-p 00000000 00:00 0 
...
...
(about 500 lines of this)
...
...
7fad9ca72000-7fad9ca73000 ---p 00000000 00:00 0 
7fad9ca73000-7fad9ca74000 rw-p 00000000 00:00 0 
7fad9ca74000-7fad9ca7c000 ---p 00000000 00:00 0 
7fad9ca7c000-7fad9cc73000 rw-p 00000000 00:00 0 
7fad9cc73000-7fad9cdda000 r-xp 00000000 08:02 73147325                   /usr/lib64/libmono-btls-shared.so
7fad9cdda000-7fad9cfda000 ---p 00167000 08:02 73147325                   /usr/lib64/libmono-btls-shared.so
7fad9cfda000-7fad9cfec000 r--p 00167000 08:02 73147325                   /usr/lib64/libmono-btls-shared.so
7fad9cfec000-7fad9cfef000 rw-p 00179000 08:02 73147325                   /usr/lib64/libmono-btls-shared.so
7fad9cfef000-7fad9cffb000 r-xp 00000000 08:02 73141856                   /usr/lib64/libnss_files-2.17.so
7fad9cffb000-7fad9d1fa000 ---p 0000c000 08:02 73141856                   /usr/lib64/libnss_files-2.17.so
7fad9d1fa000-7fad9d1fb000 r--p 0000b000 08:02 73141856                   /usr/lib64/libnss_files-2.17.so
7fad9d1fb000-7fad9d1fc000 rw-p 0000c000 08:02 73141856                   /usr/lib64/libnss_files-2.17.so
7fad9d1fc000-7fad9d202000 rw-p 00000000 00:00 0 
7fad9d202000-7fad9d2fb000 r--p 00000000 08:02 74058316                   /usr/lib/mono/gac/System.Core/4.0.0.0__b77a5c561934e089/System.Core.dll
7fad9d2fb000-7fad9d2fc000 ---p 00000000 00:00 0 
7fad9d2fc000-7fad9d2fd000 rw-p 00000000 00:00 0 
7fad9d2fd000-7fad9d305000 ---p 00000000 00:00 0 
7fad9d305000-7fad9d4fc000 rw-p 00000000 00:00 0 
7fad9d4fc000-7fad9d700000 r--p 00000000 08:02 74057786                   /usr/lib/mono/gac/System.Data/4.0.0.0__b77a5c561934e089/System.Data.dll
7fad9d700000-7fad9d802000 rw-p 00000000 00:00 0 
7fad9d802000-7fad9d80f000 rw-p 00000000 00:00 0 
7fad9d80f000-7fad9d81f000 rwxp 00000000 00:00 0 
7fad9d81f000-7fad9d83e000 r--p 00000000 08:02 74058367                   /usr/lib/mono/gac/System.Numerics/4.0.0.0__b77a5c561934e089/System.Numerics.dll
7fad9d83e000-7fad9db40000 r--p 00000000 08:02 74058408                   /usr/lib/mono/gac/System.Xml/4.0.0.0__b77a5c561934e089/System.Xml.dll
7fad9db40000-7fad9de03000 r--p 00000000 08:02 74058411                   /usr/lib/mono/gac/System/4.0.0.0__b77a5c561934e089/System.dll
7fad9de03000-7fad9dead000 r--p 00000000 08:02 11928716                   /home/user/dlls/NLog.dll
7fad9dead000-7fad9e605000 r-xp 00000000 08:02 74058222                   /usr/lib/mono/4.5/mscorlib.dll.so
7fad9e605000-7fad9e804000 ---p 00758000 08:02 74058222                   /usr/lib/mono/4.5/mscorlib.dll.so
7fad9e804000-7fad9e805000 r--p 00757000 08:02 74058222                   /usr/lib/mono/4.5/mscorlib.dll.so
7fad9e805000-7fad9e806000 rw-p 00758000 08:02 74058222                   /usr/lib/mono/4.5/mscorlib.dll.so
7fad9e806000-7fad9e82d000 rw-p 00000000 00:00 0 
7fad9e82d000-7fad9ebff000 r--p 00000000 08:02 74058221                   /usr/lib/mono/4.5/mscorlib.dll
7fad9ebff000-7fad9fbff000 rw-p 00000000 00:00 0 
7fad9fbff000-7fad9fc00000 ---p 00000000 00:00 0 
7fad9fc00000-7fada0806000 rw-p 00000000 00:00 0 
7fada0806000-7fada080d000 rw-p 00000000 00:00 0 
7fada080d000-7fada0810000 rw-p 00000000 00:00 0 
7fada0810000-7fada082c000 r--p 00000000 08:02 11928711                   /home/user/dlls/Fasterflect.dll
7fada082c000-7fada085c000 rw-p 00000000 00:00 0 
7fada085c000-7fada086e000 r--p 00000000 08:02 74058241                   /usr/lib/mono/gac/I18N.West/4.0.0.0__0738eb9f132ed756/I18N.West.dll
7fada086e000-7fada0878000 r--p 00000000 08:02 74058244                   /usr/lib/mono/gac/I18N/4.0.0.0__0738eb9f132ed756/I18N.dll
7fada0878000-7fada0888000 rw-p 00000000 00:00 0 
7fada0888000-7fada0889000 ---p 00000000 00:00 0 
7fada0889000-7fada088a000 rw-p 00000000 00:00 0 
7fada088a000-7fada0892000 ---p 00000000 00:00 0 
7fada0892000-7fada0a89000 rw-p 00000000 00:00 0 
7fada0a89000-7fada6fb2000 r--p 00000000 08:02 73141881                   /usr/lib/locale/locale-archive
7fada6fb2000-7fada7175000 r-xp 00000000 08:02 73141838                   /usr/lib64/libc-2.17.so
7fada7175000-7fada7374000 ---p 001c3000 08:02 73141838                   /usr/lib64/libc-2.17.so
7fada7374000-7fada7378000 r--p 001c2000 08:02 73141838                   /usr/lib64/libc-2.17.so
7fada7378000-7fada737a000 rw-p 001c6000 08:02 73141838                   /usr/lib64/libc-2.17.so
7fada737a000-7fada737f000 rw-p 00000000 00:00 0 
7fada737f000-7fada7394000 r-xp 00000000 08:02 73146656                   /usr/lib64/libgcc_s-4.8.5-20150702.so.1
7fada7394000-7fada7593000 ---p 00015000 08:02 73146656                   /usr/lib64/libgcc_s-4.8.5-20150702.so.1
7fada7593000-7fada7594000 r--p 00014000 08:02 73146656                   /usr/lib64/libgcc_s-4.8.5-20150702.so.1
7fada7594000-7fada7595000 rw-p 00015000 08:02 73146656                   /usr/lib64/libgcc_s-4.8.5-20150702.so.1
7fada7595000-7fada75ac000 r-xp 00000000 08:02 73141864                   /usr/lib64/libpthread-2.17.so
7fada75ac000-7fada77ab000 ---p 00017000 08:02 73141864                   /usr/lib64/libpthread-2.17.so
7fada77ab000-7fada77ac000 r--p 00016000 08:02 73141864                   /usr/lib64/libpthread-2.17.so
7fada77ac000-7fada77ad000 rw-p 00017000 08:02 73141864                   /usr/lib64/libpthread-2.17.so
7fada77ad000-7fada77b1000 rw-p 00000000 00:00 0 
7fada77b1000-7fada77b3000 r-xp 00000000 08:02 73141844                   /usr/lib64/libdl-2.17.so
7fada77b3000-7fada79b3000 ---p 00002000 08:02 73141844                   /usr/lib64/libdl-2.17.so
7fada79b3000-7fada79b4000 r--p 00002000 08:02 73141844                   /usr/lib64/libdl-2.17.so
7fada79b4000-7fada79b5000 rw-p 00003000 08:02 73141844                   /usr/lib64/libdl-2.17.so
7fada79b5000-7fada79bc000 r-xp 00000000 08:02 73141868                   /usr/lib64/librt-2.17.so
7fada79bc000-7fada7bbb000 ---p 00007000 08:02 73141868                   /usr/lib64/librt-2.17.so
7fada7bbb000-7fada7bbc000 r--p 00006000 08:02 73141868                   /usr/lib64/librt-2.17.so
7fada7bbc000-7fada7bbd000 rw-p 00007000 08:02 73141868                   /usr/lib64/librt-2.17.so
7fada7bbd000-7fada7cbe000 r-xp 00000000 08:02 73141846                   /usr/lib64/libm-2.17.so
7fada7cbe000-7fada7ebd000 ---p 00101000 08:02 73141846                   /usr/lib64/libm-2.17.so
7fada7ebd000-7fada7ebe000 r--p 00100000 08:02 73141846                   /usr/lib64/libm-2.17.so
7fada7ebe000-7fada7ebf000 rw-p 00101000 08:02 73141846                   /usr/lib64/libm-2.17.so
7fada7ebf000-7fada7ee1000 r-xp 00000000 08:02 73141831                   /usr/lib64/ld-2.17.so
7fada7ee1000-7fada7eee000 rw-p 00000000 00:00 0 
7fada7eee000-7fada7ef7000 r--p 00000000 08:02 74057806                   /usr/lib/mono/gac/System.Transactions/4.0.0.0__b77a5c561934e089/System.Transactions.dll
7fada7ef7000-7fada7f43000 r--p 00000000 08:02 74058299                   /usr/lib/mono/gac/Mono.Security/4.0.0.0__0738eb9f132ed756/Mono.Security.dll
7fada7f43000-7fada7f44000 rw-p 00000000 00:00 0 
7fada7f44000-7fada7f62000 r--p 00000000 08:02 74058312                   /usr/lib/mono/gac/System.Configuration/4.0.0.0__b03f5f7f11d50a3a/System.Configuration.dll
7fada7f62000-7fada7f7d000 r--p 00000000 08:02 11928713                   /home/user/dlls/program.exe
7fada7f7d000-7fada8074000 rw-p 00000000 00:00 0 
7fada8074000-7fada80d3000 ---p 00000000 00:00 0 
7fada80d3000-7fada80d9000 rw-p 00000000 00:00 0 
7fada80d9000-7fada80de000 rw-p 00000000 00:00 0 
7fada80de000-7fada80df000 rw-s 00000000 00:13 16976811                   /dev/shm/mono.31039
7fada80df000-7fada80e0000 rw-p 00000000 00:00 0 
7fada80e0000-7fada80e1000 r--p 00021000 08:02 73141831                   /usr/lib64/ld-2.17.so
7fada80e1000-7fada80e2000 rw-p 00022000 08:02 73141831                   /usr/lib64/ld-2.17.so
7fada80e2000-7fada80e3000 rw-p 00000000 00:00 0 
7ffe976d1000-7ffe976d9000 ---p 00000000 00:00 0 
7ffe97eaf000-7ffe97ed0000 rw-p 00000000 00:00 0                          [stack]
7ffe97fd3000-7ffe97fd5000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]

Native stacktrace:

  mono() [0x4c1273]
  /lib64/libpthread.so.0(+0xf6d0) [0x7fada75a46d0]
  /lib64/libc.so.6(gsignal+0x37) [0x7fada6fe8277]
  /lib64/libc.so.6(abort+0x148) [0x7fada6fe9968]
  mono() [0x6e1419]
  mono(monoeg_g_logv+0x4d) [0x6f5d4d]
  mono(monoeg_assertion_message+0xe0) [0x6f5f30]
  mono() [0x4c0dcb]
  mono() [0x42d2fc]
  mono() [0x42d357]
  mono() [0x628e15]
  /lib64/libpthread.so.0(+0x7e25) [0x7fada759ce25]
  /lib64/libc.so.6(clone+0x6d) [0x7fada70b0bad]

Debug info from gdb:

warning: File "/usr/bin/mono-sgen-gdb.py" auto-loading has been declined by your `auto-load safe-path' set to "$debugdir:$datadir/auto-load:/usr/bin/mono-gdb.py".
To enable execution of this file add
  add-auto-load-safe-path /usr/bin/mono-sgen-gdb.py
line to your configuration file "/root/.gdbinit".
To completely disable this security protection add
  set auto-load safe-path /
line to your configuration file "/root/.gdbinit".
For more information about this security protection see the
"Auto-loading safe path" section in the GDB manual.  E.g., run from the shell:
  info "(gdb)Auto-loading safe path"
[New LWP 15816]
[New LWP 15243]
[New LWP 15209]
[New LWP 15168]
[New LWP 15080]
[New LWP 14863]
[New LWP 14485]
[New LWP 14312]
[New LWP 13694]
[New LWP 13515]
[New LWP 13511]
[New LWP 13507]
[New LWP 12921]
[New LWP 12920]
[New LWP 12846]
[New LWP 11519]
[New LWP 10995]
[New LWP 10444]
[New LWP 10443]
[New LWP 10047]
[New LWP 8130]
[New LWP 7898]
[New LWP 31042]
[New LWP 31041]
[New LWP 31040]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
0x00007fada6fe85f2 in sigsuspend () from /lib64/libc.so.6
  Id   Target Id         Frame 
  26   Thread 0x7fada03ff700 (LWP 31040) "SGen worker" 0x00000000006b95ea in major_scan_object_concurrent_with_evacuation (full_object=<optimized out>, desc=<optimized out>, queue=queue@entry=0x7fada80da010) at sgen-scan-object.h:64
  25   Thread 0x7fada0a88700 (LWP 31041) "Finalizer" 0x00007fada75a2afb in do_futex_wait.constprop.1 () from /lib64/libpthread.so.0
  24   Thread 0x7fad9d4fb700 (LWP 31042) "Timer-Scheduler" 0x00007fada75a2afb in do_futex_wait.constprop.1 () from /lib64/libpthread.so.0
  23   Thread 0x7fad9c46e700 (LWP 7898) "Thread Pool Wor" 0x00007fada75a2d11 in do_futex_wait () from /lib64/libpthread.so.0
  22   Thread 0x7fad43dff700 (LWP 8130) "Thread Pool Wor" 0x00007fada75a2d11 in do_futex_wait () from /lib64/libpthread.so.0
  21   Thread 0x7fad7edff700 (LWP 10047) "Thread Pool Wor" 0x00007fada75a2d11 in do_futex_wait () from /lib64/libpthread.so.0
  20   Thread 0x7fad7f1ff700 (LWP 10443) "Thread Pool Wor" 0x00007fada75a2d11 in do_futex_wait () from /lib64/libpthread.so.0
  19   Thread 0x7fad7ebfe700 (LWP 10444) "Thread Pool Wor" 0x00007fada75a2d11 in do_futex_wait () from /lib64/libpthread.so.0
  18   Thread 0x7fad7fdfe700 (LWP 10995) "Thread Pool Wor" 0x00007fada75a2d11 in do_futex_wait () from /lib64/libpthread.so.0
  17   Thread 0x7fad9c66f700 (LWP 11519) "Thread Pool Wor" 0x00007fada75a2d11 in do_futex_wait () from /lib64/libpthread.so.0
  16   Thread 0x7fad7fbfd700 (LWP 12846) "Thread Pool Wor" 0x00007fada75a2d11 in do_futex_wait () from /lib64/libpthread.so.0
  15   Thread 0x7fad7f7ff700 (LWP 12920) "Thread Pool Wor" 0x00007fada75a2d11 in do_futex_wait () from /lib64/libpthread.so.0
  14   Thread 0x7fad7d1ff700 (LWP 12921) "Thread Pool Wor" 0x00007fada75a2d11 in do_futex_wait () from /lib64/libpthread.so.0
  13   Thread 0x7fad7f5fe700 (LWP 13507) "Model Id 39335" 0x00007fada75a2afb in do_futex_wait.constprop.1 () from /lib64/libpthread.so.0
  12   Thread 0x7fad9ca71700 (LWP 13511) "Model Id 40926" 0x00007fada75a2afb in do_futex_wait.constprop.1 () from /lib64/libpthread.so.0
  11   Thread 0x7fad7e3fe700 (LWP 13515) "Model Id 43312" 0x00007fada75a2afb in do_futex_wait.constprop.1 () from /lib64/libpthread.so.0
  10   Thread 0x7fad7dbff700 (LWP 13694) "Thread Pool Wor" 0x00007fada75a2d11 in do_futex_wait () from /lib64/libpthread.so.0
  9    Thread 0x7fad7d9fe700 (LWP 14312) "Model Id 45699" 0x00007fada75a351d in __lll_lock_wait () from /lib64/libpthread.so.0
  8    Thread 0x7fac7786b700 (LWP 14485) "Thread Pool Wor" 0x00007fada75a2afb in do_futex_wait.constprop.1 () from /lib64/libpthread.so.0
  7    Thread 0x7fad9c26d700 (LWP 14863) "Thread Pool Wor" 0x00007fada75a2d11 in do_futex_wait () from /lib64/libpthread.so.0
  6    Thread 0x7fad7ffff700 (LWP 15080) "Model Id 47290" 0x00007fada75a2afb in do_futex_wait.constprop.1 () from /lib64/libpthread.so.0
  5    Thread 0x7fad9c870700 (LWP 15168) "Model Id 48881" 0x00007fada75a2afb in do_futex_wait.constprop.1 () from /lib64/libpthread.so.0
  4    Thread 0x7fac773d8700 (LWP 15209) "Model Id 50472" 0x00007fada75a2afb in do_futex_wait.constprop.1 () from /lib64/libpthread.so.0
  3    Thread 0x7fad9cc72700 (LWP 15243) "mono" 0x00007fada75a2afb in do_futex_wait.constprop.1 () from /lib64/libpthread.so.0
  2    Thread 0x7fad7c9ff700 (LWP 15816) "Timer-Scheduler" 0x00007fada75a4279 in waitpid () from /lib64/libpthread.so.0
* 1    Thread 0x7fada80d3780 (LWP 31039) "mono" 0x00007fada6fe85f2 in sigsuspend () from /lib64/libc.so.6

Thread 26 (Thread 0x7fada03ff700 (LWP 31040)):
#0  0x00000000006b95ea in major_scan_object_concurrent_with_evacuation (full_object=<optimized out>, desc=<optimized out>, queue=queue@entry=0x7fada80da010) at sgen-scan-object.h:64
#1  0x00000000006bbfed in drain_gray_stack_concurrent_with_evacuation (queue=<optimized out>) at sgen-marksweep-drain-gray-stack.h:345
#2  drain_gray_stack_concurrent (queue=0x7fada80da010) at sgen-marksweep.c:1346
#3  0x00000000006d890d in marker_idle_func (data_untyped=0x7fada80da008) at sgen-workers.c:337
#4  0x00000000006d71f7 in thread_func (data=<optimized out>) at sgen-thread-pool.c:222
#5  0x00007fada759ce25 in start_thread () from /lib64/libpthread.so.0
#6  0x00007fada70b0bad in clone () from /lib64/libc.so.6

Thread 25 (Thread 0x7fada0a88700 (LWP 31041)):
#0  0x00007fada75a2afb in do_futex_wait.constprop.1 () from /lib64/libpthread.so.0
#1  0x00007fada75a2b8f in __new_sem_wait_slow.constprop.0 () from /lib64/libpthread.so.0
#2  0x00007fada75a2c2b in sem_wait@@GLIBC_2.2.5 () from /lib64/libpthread.so.0
#3  0x000000000067110c in mono_os_sem_wait (flags=MONO_SEM_FLAGS_ALERTABLE, sem=0xa34b40 <finalizer_sem>) at ../../mono/utils/mono-os-semaphore.h:209
#4  mono_coop_sem_wait (flags=MONO_SEM_FLAGS_ALERTABLE, sem=0xa34b40 <finalizer_sem>) at ../../mono/utils/mono-coop-semaphore.h:43
#5  finalizer_thread (unused=unused@entry=0x0) at gc.c:903
#6  0x0000000000628e96 in start_wrapper_internal (stack_ptr=<optimized out>, start_info=0x0) at threads.c:1098
#7  start_wrapper (data=0x129ff50) at threads.c:1158
#8  0x00007fada759ce25 in start_thread () from /lib64/libpthread.so.0
#9  0x00007fada70b0bad in clone () from /lib64/libc.so.6

Thread 24 (Thread 0x7fad9d4fb700 (LWP 31042)):
#0  0x00007fada75a2afb in do_futex_wait.constprop.1 () from /lib64/libpthread.so.0
#1  0x00007fada75a2b8f in __new_sem_wait_slow.constprop.0 () from /lib64/libpthread.so.0
#2  0x00007fada75a2c2b in sem_wait@@GLIBC_2.2.5 () from /lib64/libpthread.so.0
#3  0x0000000000626d34 in mono_os_sem_wait (flags=MONO_SEM_FLAGS_NONE, sem=0x7fad90003df8) at ../../mono/utils/mono-os-semaphore.h:209
#4  mono_coop_sem_wait (flags=MONO_SEM_FLAGS_NONE, sem=0x7fad90003df8) at ../../mono/utils/mono-coop-semaphore.h:43
#5  create_thread (thread=thread@entry=0x7fabed296de0, internal=internal@entry=0x7fa9d39ef328, start_delegate=start_delegate@entry=0x0, start_func=start_func@entry=0x680d20 <monitor_thread>, start_func_arg=start_func_arg@entry=0x0, flags=flags@entry=(MONO_THREAD_CREATE_FLAGS_THREADPOOL | MONO_THREAD_CREATE_FLAGS_SMALL_STACK), error=error@entry=0x7fad9d4fa680) at threads.c:1258
#6  0x000000000062722f in mono_thread_create_internal (domain=<optimized out>, func=func@entry=0x680d20 <monitor_thread>, arg=arg@entry=0x0, flags=flags@entry=(MONO_THREAD_CREATE_FLAGS_THREADPOOL | MONO_THREAD_CREATE_FLAGS_SMALL_STACK), error=error@entry=0x7fad9d4fa680) at threads.c:1322
#7  0x0000000000680c44 in monitor_ensure_running () at threadpool-worker-default.c:785
#8  worker_request () at threadpool-worker-default.c:590
#9  0x0000000000681578 in mono_threadpool_worker_request () at threadpool-worker-default.c:354
#10 0x000000000062b7f5 in ves_icall_System_Threading_ThreadPool_RequestWorkerThread (error=<optimized out>) at threadpool.c:801
#11 0x0000000041a62963 in ?? ()
#12 0x00000000ffffffff in ?? ()
#13 0x0000000000000000 in ?? ()

Thread 23 (Thread 0x7fad9c46e700 (LWP 7898)):
#0  0x00007fada75a2d11 in do_futex_wait () from /lib64/libpthread.so.0
#1  0x00007fada75a2dd7 in __new_sem_wait_slow () from /lib64/libpthread.so.0
#2  0x00007fada75a2e75 in sem_timedwait () from /lib64/libpthread.so.0
#3  0x0000000000680210 in mono_os_sem_timedwait (flags=MONO_SEM_FLAGS_ALERTABLE, timeout_ms=<optimized out>, sem=0xa35448 <worker+72>) at ../../mono/utils/mono-os-semaphore.h:258
#4  mono_coop_sem_timedwait (sem=0xa35448 <worker+72>, flags=MONO_SEM_FLAGS_ALERTABLE, timeout_ms=<optimized out>) at ../../mono/utils/mono-coop-semaphore.h:57
#5  worker_park () at threadpool-worker-default.c:391
#6  worker_thread (unused=unused@entry=0x0) at threadpool-worker-default.c:483
#7  0x0000000000628e96 in start_wrapper_internal (stack_ptr=<optimized out>, start_info=0x0) at threads.c:1098
#8  start_wrapper (data=0x7fad6c002e00) at threads.c:1158
#9  0x00007fada759ce25 in start_thread () from /lib64/libpthread.so.0
#10 0x00007fada70b0bad in clone () from /lib64/libc.so.6

Thread 22 (Thread 0x7fad43dff700 (LWP 8130)):
#0  0x00007fada75a2d11 in do_futex_wait () from /lib64/libpthread.so.0
#1  0x00007fada75a2dd7 in __new_sem_wait_slow () from /lib64/libpthread.so.0
#2  0x00007fada75a2e75 in sem_timedwait () from /lib64/libpthread.so.0
#3  0x0000000000680210 in mono_os_sem_timedwait (flags=MONO_SEM_FLAGS_ALERTABLE, timeout_ms=<optimized out>, sem=0xa35448 <worker+72>) at ../../mono/utils/mono-os-semaphore.h:258
#4  mono_coop_sem_timedwait (sem=0xa35448 <worker+72>, flags=MONO_SEM_FLAGS_ALERTABLE, timeout_ms=<optimized out>) at ../../mono/utils/mono-coop-semaphore.h:57
#5  worker_park () at threadpool-worker-default.c:391
#6  worker_thread (unused=unused@entry=0x0) at threadpool-worker-default.c:483
#7  0x0000000000628e96 in start_wrapper_internal (stack_ptr=<optimized out>, start_info=0x0) at threads.c:1098
#8  start_wrapper (data=0x7fad440019b0) at threads.c:1158
#9  0x00007fada759ce25 in start_thread () from /lib64/libpthread.so.0
#10 0x00007fada70b0bad in clone () from /lib64/libc.so.6

Thread 21 (Thread 0x7fad7edff700 (LWP 10047)):
#0  0x00007fada75a2d11 in do_futex_wait () from /lib64/libpthread.so.0
#1  0x00007fada75a2dd7 in __new_sem_wait_slow () from /lib64/libpthread.so.0
#2  0x00007fada75a2e75 in sem_timedwait () from /lib64/libpthread.so.0
#3  0x0000000000680210 in mono_os_sem_timedwait (flags=MONO_SEM_FLAGS_ALERTABLE, timeout_ms=<optimized out>, sem=0xa35448 <worker+72>) at ../../mono/utils/mono-os-semaphore.h:258
#4  mono_coop_sem_timedwait (sem=0xa35448 <worker+72>, flags=MONO_SEM_FLAGS_ALERTABLE, timeout_ms=<optimized out>) at ../../mono/utils/mono-coop-semaphore.h:57
#5  worker_park () at threadpool-worker-default.c:391
#6  worker_thread (unused=unused@entry=0x0) at threadpool-worker-default.c:483
#7  0x0000000000628e96 in start_wrapper_internal (stack_ptr=<optimized out>, start_info=0x0) at threads.c:1098
#8  start_wrapper (data=0x7fad48002d30) at threads.c:1158
#9  0x00007fada759ce25 in start_thread () from /lib64/libpthread.so.0
#10 0x00007fada70b0bad in clone () from /lib64/libc.so.6

Thread 20 (Thread 0x7fad7f1ff700 (LWP 10443)):
#0  0x00007fada75a2d11 in do_futex_wait () from /lib64/libpthread.so.0
#1  0x00007fada75a2dd7 in __new_sem_wait_slow () from /lib64/libpthread.so.0
#2  0x00007fada75a2e75 in sem_timedwait () from /lib64/libpthread.so.0
#3  0x0000000000680210 in mono_os_sem_timedwait (flags=MONO_SEM_FLAGS_ALERTABLE, timeout_ms=<optimized out>, sem=0xa35448 <worker+72>) at ../../mono/utils/mono-os-semaphore.h:258
#4  mono_coop_sem_timedwait (sem=0xa35448 <worker+72>, flags=MONO_SEM_FLAGS_ALERTABLE, timeout_ms=<optimized out>) at ../../mono/utils/mono-coop-semaphore.h:57
#5  worker_park () at threadpool-worker-default.c:391
#6  worker_thread (unused=unused@entry=0x0) at threadpool-worker-default.c:483
#7  0x0000000000628e96 in start_wrapper_internal (stack_ptr=<optimized out>, start_info=0x0) at threads.c:1098
#8  start_wrapper (data=0x7fad7029e8d0) at threads.c:1158
#9  0x00007fada759ce25 in start_thread () from /lib64/libpthread.so.0
#10 0x00007fada70b0bad in clone () from /lib64/libc.so.6

Thread 19 (Thread 0x7fad7ebfe700 (LWP 10444)):
#0  0x00007fada75a2d11 in do_futex_wait () from /lib64/libpthread.so.0
#1  0x00007fada75a2dd7 in __new_sem_wait_slow () from /lib64/libpthread.so.0
#2  0x00007fada75a2e75 in sem_timedwait () from /lib64/libpthread.so.0
#3  0x0000000000680210 in mono_os_sem_timedwait (flags=MONO_SEM_FLAGS_ALERTABLE, timeout_ms=<optimized out>, sem=0xa35448 <worker+72>) at ../../mono/utils/mono-os-semaphore.h:258
#4  mono_coop_sem_timedwait (sem=0xa35448 <worker+72>, flags=MONO_SEM_FLAGS_ALERTABLE, timeout_ms=<optimized out>) at ../../mono/utils/mono-coop-semaphore.h:57
#5  worker_park () at threadpool-worker-default.c:391
#6  worker_thread (unused=unused@entry=0x0) at threadpool-worker-default.c:483
#7  0x0000000000628e96 in start_wrapper_internal (stack_ptr=<optimized out>, start_info=0x0) at threads.c:1098
#8  start_wrapper (data=0x7fad50003700) at threads.c:1158
#9  0x00007fada759ce25 in start_thread () from /lib64/libpthread.so.0
#10 0x00007fada70b0bad in clone () from /lib64/libc.so.6

Thread 18 (Thread 0x7fad7fdfe700 (LWP 10995)):
#0  0x00007fada75a2d11 in do_futex_wait () from /lib64/libpthread.so.0
#1  0x00007fada75a2dd7 in __new_sem_wait_slow () from /lib64/libpthread.so.0
#2  0x00007fada75a2e75 in sem_timedwait () from /lib64/libpthread.so.0
#3  0x0000000000680210 in mono_os_sem_timedwait (flags=MONO_SEM_FLAGS_ALERTABLE, timeout_ms=<optimized out>, sem=0xa35448 <worker+72>) at ../../mono/utils/mono-os-semaphore.h:258
#4  mono_coop_sem_timedwait (sem=0xa35448 <worker+72>, flags=MONO_SEM_FLAGS_ALERTABLE, timeout_ms=<optimized out>) at ../../mono/utils/mono-coop-semaphore.h:57
#5  worker_park () at threadpool-worker-default.c:391
#6  worker_thread (unused=unused@entry=0x0) at threadpool-worker-default.c:483
#7  0x0000000000628e96 in start_wrapper_internal (stack_ptr=<optimized out>, start_info=0x0) at threads.c:1098
#8  start_wrapper (data=0x7fad60004720) at threads.c:1158
#9  0x00007fada759ce25 in start_thread () from /lib64/libpthread.so.0
#10 0x00007fada70b0bad in clone () from /lib64/libc.so.6

Thread 17 (Thread 0x7fad9c66f700 (LWP 11519)):
#0  0x00007fada75a2d11 in do_futex_wait () from /lib64/libpthread.so.0
#1  0x00007fada75a2dd7 in __new_sem_wait_slow () from /lib64/libpthread.so.0
#2  0x00007fada75a2e75 in sem_timedwait () from /lib64/libpthread.so.0
#3  0x0000000000680210 in mono_os_sem_timedwait (flags=MONO_SEM_FLAGS_ALERTABLE, timeout_ms=<optimized out>, sem=0xa35448 <worker+72>) at ../../mono/utils/mono-os-semaphore.h:258
#4  mono_coop_sem_timedwait (sem=0xa35448 <worker+72>, flags=MONO_SEM_FLAGS_ALERTABLE, timeout_ms=<optimized out>) at ../../mono/utils/mono-coop-semaphore.h:57
#5  worker_park () at threadpool-worker-default.c:391
#6  worker_thread (unused=unused@entry=0x0) at threadpool-worker-default.c:483
#7  0x0000000000628e96 in start_wrapper_internal (stack_ptr=<optimized out>, start_info=0x0) at threads.c:1098
#8  start_wrapper (data=0x7fad54008560) at threads.c:1158
#9  0x00007fada759ce25 in start_thread () from /lib64/libpthread.so.0
#10 0x00007fada70b0bad in clone () from /lib64/libc.so.6

Thread 16 (Thread 0x7fad7fbfd700 (LWP 12846)):
#0  0x00007fada75a2d11 in do_futex_wait () from /lib64/libpthread.so.0
#1  0x00007fada75a2dd7 in __new_sem_wait_slow () from /lib64/libpthread.so.0
#2  0x00007fada75a2e75 in sem_timedwait () from /lib64/libpthread.so.0
#3  0x0000000000680210 in mono_os_sem_timedwait (flags=MONO_SEM_FLAGS_ALERTABLE, timeout_ms=<optimized out>, sem=0xa35448 <worker+72>) at ../../mono/utils/mono-os-semaphore.h:258
#4  mono_coop_sem_timedwait (sem=0xa35448 <worker+72>, flags=MONO_SEM_FLAGS_ALERTABLE, timeout_ms=<optimized out>) at ../../mono/utils/mono-coop-semaphore.h:57
#5  worker_park () at threadpool-worker-default.c:391
#6  worker_thread (unused=unused@entry=0x0) at threadpool-worker-default.c:483
#7  0x0000000000628e96 in start_wrapper_internal (stack_ptr=<optimized out>, start_info=0x0) at threads.c:1098
#8  start_wrapper (data=0x7fad540084e0) at threads.c:1158
#9  0x00007fada759ce25 in start_thread () from /lib64/libpthread.so.0
#10 0x00007fada70b0bad in clone () from /lib64/libc.so.6

Thread 15 (Thread 0x7fad7f7ff700 (LWP 12920)):
#0  0x00007fada75a2d11 in do_futex_wait () from /lib64/libpthread.so.0
#1  0x00007fada75a2dd7 in __new_sem_wait_slow () from /lib64/libpthread.so.0
#2  0x00007fada75a2e75 in sem_timedwait () from /lib64/libpthread.so.0
#3  0x0000000000680210 in mono_os_sem_timedwait (flags=MONO_SEM_FLAGS_ALERTABLE, timeout_ms=<optimized out>, sem=0xa35448 <worker+72>) at ../../mono/utils/mono-os-semaphore.h:258
#4  mono_coop_sem_timedwait (sem=0xa35448 <worker+72>, flags=MONO_SEM_FLAGS_ALERTABLE, timeout_ms=<optimized out>) at ../../mono/utils/mono-coop-semaphore.h:57
#5  worker_park () at threadpool-worker-default.c:391
#6  worker_thread (unused=unused@entry=0x0) at threadpool-worker-default.c:483
#7  0x0000000000628e96 in start_wrapper_internal (stack_ptr=<optimized out>, start_info=0x0) at threads.c:1098
#8  start_wrapper (data=0x7fad5c004920) at threads.c:1158
#9  0x00007fada759ce25 in start_thread () from /lib64/libpthread.so.0
#10 0x00007fada70b0bad in clone () from /lib64/libc.so.6

Thread 14 (Thread 0x7fad7d1ff700 (LWP 12921)):
#0  0x00007fada75a2d11 in do_futex_wait () from /lib64/libpthread.so.0
#1  0x00007fada75a2dd7 in __new_sem_wait_slow () from /lib64/libpthread.so.0
#2  0x00007fada75a2e75 in sem_timedwait () from /lib64/libpthread.so.0
#3  0x0000000000680210 in mono_os_sem_timedwait (flags=MONO_SEM_FLAGS_ALERTABLE, timeout_ms=<optimized out>, sem=0xa35448 <worker+72>) at ../../mono/utils/mono-os-semaphore.h:258
#4  mono_coop_sem_timedwait (sem=0xa35448 <worker+72>, flags=MONO_SEM_FLAGS_ALERTABLE, timeout_ms=<optimized out>) at ../../mono/utils/mono-coop-semaphore.h:57
#5  worker_park () at threadpool-worker-default.c:391
#6  worker_thread (unused=unused@entry=0x0) at threadpool-worker-default.c:483
#7  0x0000000000628e96 in start_wrapper_internal (stack_ptr=<optimized out>, start_info=0x0) at threads.c:1098
#8  start_wrapper (data=0x7fad60004830) at threads.c:1158
#9  0x00007fada759ce25 in start_thread () from /lib64/libpthread.so.0
#10 0x00007fada70b0bad in clone () from /lib64/libc.so.6

Thread 13 (Thread 0x7fad7f5fe700 (LWP 13507)):
#0  0x00007fada75a2afb in do_futex_wait.constprop.1 () from /lib64/libpthread.so.0
#1  0x00007fada75a2b8f in __new_sem_wait_slow.constprop.0 () from /lib64/libpthread.so.0
#2  0x00007fada75a2c2b in sem_wait@@GLIBC_2.2.5 () from /lib64/libpthread.so.0
#3  0x000000000042abfb in mono_os_sem_wait (flags=<optimized out>, sem=<optimized out>) at ../../mono/utils/mono-os-semaphore.h:209
#4  mono_os_sem_timedwait (flags=MONO_SEM_FLAGS_ALERTABLE, timeout_ms=4294967295, sem=0x7fad700085b0) at ../../mono/utils/mono-os-semaphore.h:242
#5  mono_coop_sem_timedwait (sem=0x7fad700085b0, timeout_ms=timeout_ms@entry=4294967295, flags=MONO_SEM_FLAGS_ALERTABLE) at ../../mono/utils/mono-coop-semaphore.h:57
#6  0x000000000067319b in mono_monitor_try_enter_inflated (obj=0x7fada0400718, ms=<optimized out>, allow_interruption=1, id=<optimized out>) at monitor.c:907
#7  0x000000000067478e in mono_monitor_try_enter_internal (allow_interruption=<optimized out>, ms=<optimized out>, obj=<optimized out>) at monitor.c:993
#8  ves_icall_System_Threading_Monitor_Monitor_try_enter_with_atomic_var (obj=<optimized out>, ms=<optimized out>, lockTaken=<optimized out>) at monitor.c:1156
#9  0x0000000041a9bfc7 in ?? ()
#10 0x0000000000000001 in ?? ()
#11 0x00007fada048c278 in ?? ()
#12 0x00007fad7f5fce58 in ?? ()
#13 0x00007fada0400718 in ?? ()
#14 0x00007fada04ad8f8 in ?? ()
#15 0x00007fad68018d90 in ?? ()
#16 0x00007fad7f5fcfd0 in ?? ()
#17 0x00007fad7f5fccf0 in ?? ()
#18 0x00007fada048c278 in ?? ()
#19 0x0000000041a27018 in ?? ()
#20 0x000000000000006d in ?? ()
#21 0x0000000000000000 in ?? ()

Thread 12 (Thread 0x7fad9ca71700 (LWP 13511)):
#0  0x00007fada75a2afb in do_futex_wait.constprop.1 () from /lib64/libpthread.so.0
#1  0x00007fada75a2b8f in __new_sem_wait_slow.constprop.0 () from /lib64/libpthread.so.0
#2  0x00007fada75a2c2b in sem_wait@@GLIBC_2.2.5 () from /lib64/libpthread.so.0
#3  0x000000000042abfb in mono_os_sem_wait (flags=<optimized out>, sem=<optimized out>) at ../../mono/utils/mono-os-semaphore.h:209
#4  mono_os_sem_timedwait (flags=MONO_SEM_FLAGS_ALERTABLE, timeout_ms=4294967295, sem=0x7fad700085b0) at ../../mono/utils/mono-os-semaphore.h:242
#5  mono_coop_sem_timedwait (sem=0x7fad700085b0, timeout_ms=timeout_ms@entry=4294967295, flags=MONO_SEM_FLAGS_ALERTABLE) at ../../mono/utils/mono-coop-semaphore.h:57
#6  0x000000000067319b in mono_monitor_try_enter_inflated (obj=0x7fada0400718, ms=<optimized out>, allow_interruption=1, id=<optimized out>) at monitor.c:907
#7  0x000000000067478e in mono_monitor_try_enter_internal (allow_interruption=<optimized out>, ms=<optimized out>, obj=<optimized out>) at monitor.c:993
#8  ves_icall_System_Threading_Monitor_Monitor_try_enter_with_atomic_var (obj=<optimized out>, ms=<optimized out>, lockTaken=<optimized out>) at monitor.c:1156
#9  0x0000000041a9bfc7 in ?? ()
#10 0x0000000000000001 in ?? ()
#11 0x00007fada06a1b60 in ?? ()
#12 0x00007fad9ca6fe58 in ?? ()
#13 0x00007fada0400718 in ?? ()
#14 0x00007fada04ad8f8 in ?? ()
#15 0x00007fad88003460 in ?? ()
#16 0x00007fad9ca6ffd0 in ?? ()
#17 0x00007fad9ca6fcf0 in ?? ()
#18 0x00007fada06a1b60 in ?? ()
#19 0x0000000041a27018 in ?? ()
#20 0x000000000000006d in ?? ()
#21 0x0000000000000000 in ?? ()

Thread 11 (Thread 0x7fad7e3fe700 (LWP 13515)):
#0  0x00007fada75a2afb in do_futex_wait.constprop.1 () from /lib64/libpthread.so.0
#1  0x00007fada75a2b8f in __new_sem_wait_slow.constprop.0 () from /lib64/libpthread.so.0
#2  0x00007fada75a2c2b in sem_wait@@GLIBC_2.2.5 () from /lib64/libpthread.so.0
#3  0x00000000006ebb2c in mono_os_sem_wait (flags=<optimized out>, sem=<optimized out>) at ../../mono/utils/mono-os-semaphore.h:209
#4  mono_os_sem_timedwait (sem=0xa44500 <suspend_semaphore>, flags=MONO_SEM_FLAGS_NONE, timeout_ms=4294967295) at ../../mono/utils/mono-os-semaphore.h:242
#5  mono_threads_wait_pending_operations () at mono-threads.c:246
#6  0x0000000000699419 in sgen_unified_suspend_restart_world () at sgen-stw.c:405
#7  sgen_client_restart_world (generation=generation@entry=0, serial_collection=serial_collection@entry=0, stw_time=stw_time@entry=0x7fad7e3fc160) at sgen-stw.c:167
#8  0x00000000006a94a4 in sgen_restart_world (generation=generation@entry=0, serial_collection=0) at sgen-gc.c:3839
#9  0x00000000006a962e in sgen_perform_collection_inner (requested_size=requested_size@entry=4128, generation_to_collect=<optimized out>, reason=<optimized out>, forced_serial=forced_serial@entry=0, stw=stw@entry=1) at sgen-gc.c:2594
#10 0x00000000006a9b47 in sgen_perform_collection (stw=1, forced_serial=0, reason=<optimized out>, generation_to_collect=<optimized out>, requested_size=4128) at sgen-gc.c:2635
#11 sgen_ensure_free_space (size=size@entry=4128, generation=generation@entry=0) at sgen-gc.c:2509
#12 0x0000000000699a9a in sgen_alloc_obj_nolock (vtable=vtable@entry=0x12b0800, size=size@entry=4128) at sgen-alloc.c:239
#13 0x00000000006754c3 in mono_gc_alloc_vector (vtable=0x12b0800, size=4128, max_length=512) at sgen-mono.c:1310
#14 0x00000000419ef66d in ?? ()
#15 0x00007fada07ffa20 in ?? ()
#16 0x00007fad4c000f28 in ?? ()
#17 0x0000000000001020 in ?? ()
#18 0x00000000012b0800 in ?? ()
#19 0x00007fad4c0008c0 in ?? ()
#20 0x00007fad4c003bd0 in ?? ()
#21 0x00007fad7e3fc430 in ?? ()
#22 0x00007fad7e3fc380 in ?? ()
#23 0x0000000000000006 in ?? ()
#24 0x00000000419d669c in ?? ()
#25 0x0000000000000200 in ?? ()
#26 0x00007fada07fede0 in ?? ()
#27 0x00007fada0800a40 in ?? ()
#28 0x00007fada04036a8 in ?? ()
#29 0x00007fada0520210 in ?? ()
#30 0x0000000000000000 in ?? ()

Thread 10 (Thread 0x7fad7dbff700 (LWP 13694)):
#0  0x00007fada75a2d11 in do_futex_wait () from /lib64/libpthread.so.0
#1  0x00007fada75a2dd7 in __new_sem_wait_slow () from /lib64/libpthread.so.0
#2  0x00007fada75a2e75 in sem_timedwait () from /lib64/libpthread.so.0
#3  0x0000000000680210 in mono_os_sem_timedwait (flags=MONO_SEM_FLAGS_ALERTABLE, timeout_ms=<optimized out>, sem=0xa35448 <worker+72>) at ../../mono/utils/mono-os-semaphore.h:258
#4  mono_coop_sem_timedwait (sem=0xa35448 <worker+72>, flags=MONO_SEM_FLAGS_ALERTABLE, timeout_ms=<optimized out>) at ../../mono/utils/mono-coop-semaphore.h:57
#5  worker_park () at threadpool-worker-default.c:391
#6  worker_thread (unused=unused@entry=0x0) at threadpool-worker-default.c:483
#7  0x0000000000628e96 in start_wrapper_internal (stack_ptr=<optimized out>, start_info=0x0) at threads.c:1098
#8  start_wrapper (data=0x7fad702b0360) at threads.c:1158
#9  0x00007fada759ce25 in start_thread () from /lib64/libpthread.so.0
#10 0x00007fada70b0bad in clone () from /lib64/libc.so.6

Thread 9 (Thread 0x7fad7d9fe700 (LWP 14312)):
#0  0x00007fada75a351d in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00007fada759ee1b in _L_lock_812 () from /lib64/libpthread.so.0
#2  0x00007fada759ece8 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x00000000006a92a1 in mono_os_mutex_lock (mutex=0xa45620 <sgen_gc_mutex>) at ../../mono/utils/mono-os-mutex.h:99
#4  mono_coop_mutex_lock (mutex=0xa45620 <sgen_gc_mutex>) at ../../mono/utils/mono-coop-mutex.h:56
#5  sgen_gc_lock () at sgen-gc.c:3757
#6  0x00000000006754b8 in mono_gc_alloc_vector (vtable=0x132ab38, size=8032, max_length=8000) at sgen-mono.c:1308
#7  0x00000000419ef66d in ?? ()
#8  0x00007fac6f9d7150 in ?? ()
#9  0x00007fad7d9fc868 in ?? ()
#10 0x0000000000001f60 in ?? ()
#11 0x000000000132ab38 in ?? ()
#12 0x00007fad440008c0 in ?? ()
#13 0x00007fad440039d0 in ?? ()
#14 0x00007fad7d9fc7d0 in ?? ()
#15 0x00007fad7d9fc720 in ?? ()
#16 0x40669bd70a3d7001 in ?? ()
#17 0x00000000419d669c in ?? ()
#18 0x0000000000001f40 in ?? ()
#19 0x00007fad7d9fc710 in ?? ()
#20 0x00007fad7d9fc7d0 in ?? ()
#21 0x0000000041a4a404 in ?? ()
#22 0x00007fad7d9fc7c0 in ?? ()
#23 0xb947ae147ae10840 in ?? ()
#24 0xd100000001044066 in ?? ()
#25 0x00007fac6f9d7150 in ?? ()
#26 0x00007fad7d9fc868 in ?? ()
#27 0x00007fac6f9d7150 in ?? ()
#28 0x00007fada0408490 in ?? ()
#29 0x0000000000001f40 in ?? ()
#30 0x00007fad7d9fc960 in ?? ()
#31 0x0000000041a53b73 in ?? ()
#32 0x0000000000000005 in ?? ()
#33 0x00000000005b8d80 in mono_exception_from_token_two_strings_checked (image=<optimized out>, token=<optimized out>, a1=0x7fad7d9fc7d0, a2=0x7fad44000f90, error=0x7fac6f9d7150) at exception.c:377
#34 0x0001020304050607 in ?? ()
#35 0x00007fad7d9fc9b0 in ?? ()
#36 0x00007fada0408490 in ?? ()
#37 0x00007fad7d9fc9b0 in ?? ()
#38 0x00007fac6f9d7150 in ?? ()
#39 0x5724285ae2d1cda1 in ?? ()
#40 0x0851dbd897dd0dfc in ?? ()
#41 0xabac623500000000 in ?? ()
#42 0x0000000000000000 in ?? ()

Thread 8 (Thread 0x7fac7786b700 (LWP 14485)):
#0  0x00007fada75a2afb in do_futex_wait.constprop.1 () from /lib64/libpthread.so.0
#1  0x00007fada75a2b8f in __new_sem_wait_slow.constprop.0 () from /lib64/libpthread.so.0
#2  0x00007fada75a2c2b in sem_wait@@GLIBC_2.2.5 () from /lib64/libpthread.so.0
#3  0x000000000042abfb in mono_os_sem_wait (flags=<optimized out>, sem=<optimized out>) at ../../mono/utils/mono-os-semaphore.h:209
#4  mono_os_sem_timedwait (flags=MONO_SEM_FLAGS_ALERTABLE, timeout_ms=4294967295, sem=0x7fad90003670) at ../../mono/utils/mono-os-semaphore.h:242
#5  mono_coop_sem_timedwait (sem=0x7fad90003670, timeout_ms=timeout_ms@entry=4294967295, flags=MONO_SEM_FLAGS_ALERTABLE) at ../../mono/utils/mono-coop-semaphore.h:57
#6  0x000000000067319b in mono_monitor_try_enter_inflated (obj=0x7fada04ae5e8, ms=<optimized out>, allow_interruption=1, id=<optimized out>) at monitor.c:907
#7  0x000000000067478e in mono_monitor_try_enter_internal (allow_interruption=<optimized out>, ms=<optimized out>, obj=<optimized out>) at monitor.c:993
#8  ves_icall_System_Threading_Monitor_Monitor_try_enter_with_atomic_var (obj=<optimized out>, ms=<optimized out>, lockTaken=<optimized out>) at monitor.c:1156
#9  0x0000000041a9bfc7 in ?? ()
#10 0x00000000015b45b0 in ?? ()
#11 0x00007fac7786a798 in ?? ()
#12 0x00007fada04ae5e8 in ?? ()
#13 0x00007faa06962340 in ?? ()
#14 0x00007faa06962340 in ?? ()
#15 0x00007fabd4002590 in ?? ()
#16 0x00007fac7786a7c0 in ?? ()
#17 0x00007fac7786a710 in ?? ()
#18 0x00007fac7786a798 in ?? ()
#19 0x00007fad9e11f8a5 in System_Threading_Timer_Scheduler_Remove_System_Threading_Timer (this=..., timer=...) at /builddir/build/BUILD/mono-5.14.0.177/mcs/class/corlib/System.Threading/Timer.cs:305
#20 0x00007fad9e11ee40 in System_Threading_Timer_Dispose (this=...) at /builddir/build/BUILD/mono-5.14.0.177/mcs/class/corlib/System.Threading/Timer.cs:143
#21 0x0000000041a91e93 in ?? ()
#22 0x000000000137f1b0 in ?? ()
#23 0x00007faa06962340 in ?? ()
#24 0x00007fac7786a8e8 in ?? ()
#25 0x00007fad9e1060b6 in System_Threading_CancellationTokenSource_NotifyCancellation_bool (this=..., throwOnFirstException=false) at /builddir/build/BUILD/mono-5.14.0.177/mcs/class/referencesource/mscorlib/system/threading/CancellationTokenSource.cs:749
#26 0x00007fad9e10541b in System_Threading_CancellationTokenSource_Cancel_bool (this=..., throwOnFirstException=false) at /builddir/build/BUILD/mono-5.14.0.177/mcs/class/referencesource/mscorlib/system/threading/CancellationTokenSource.cs:397
#27 0x00007fad9e1053bf in System_Threading_CancellationTokenSource_Cancel (this=...) at /builddir/build/BUILD/mono-5.14.0.177/mcs/class/referencesource/mscorlib/system/threading/CancellationTokenSource.cs:362
#28 0x00007fad9e1056a8 in System_Threading_CancellationTokenSource_TimerCallbackLogic_object (obj=0x7faa06962480) at /builddir/build/BUILD/mono-5.14.0.177/mcs/class/referencesource/mscorlib/system/threading/CancellationTokenSource.cs:526
#29 0x00007fad9e12091f in System_Threading_Timer_Scheduler_TimerCB_object (o=0x7faa06962480) at /builddir/build/BUILD/mono-5.14.0.177/mcs/class/corlib/System.Threading/Timer.cs:405
#30 0x00007fad9e11808f in System_Threading_QueueUserWorkItemCallback_System_Threading_IThreadPoolWorkItem_ExecuteWorkItem (this=...) at /builddir/build/BUILD/mono-5.14.0.177/mcs/class/referencesource/mscorlib/system/threading/threadpool.cs:1281
#31 0x00007fad9e116445 in System_Threading_ThreadPoolWorkQueue_Dispatch () at /builddir/build/BUILD/mono-5.14.0.177/mcs/class/referencesource/mscorlib/system/threading/threadpool.cs:858
#32 0x00007fad9e117e89 in System_Threading__ThreadPoolWaitCallback_PerformWaitCallback () at /builddir/build/BUILD/mono-5.14.0.177/mcs/class/referencesource/mscorlib/system/threading/threadpool.cs:1213
#33 0x0000000000000000 in ?? ()

Thread 7 (Thread 0x7fad9c26d700 (LWP 14863)):
#0  0x00007fada75a2d11 in do_futex_wait () from /lib64/libpthread.so.0
#1  0x00007fada75a2dd7 in __new_sem_wait_slow () from /lib64/libpthread.so.0
#2  0x00007fada75a2e75 in sem_timedwait () from /lib64/libpthread.so.0
#3  0x0000000000680210 in mono_os_sem_timedwait (flags=MONO_SEM_FLAGS_ALERTABLE, timeout_ms=<optimized out>, sem=0xa35448 <worker+72>) at ../../mono/utils/mono-os-semaphore.h:258
#4  mono_coop_sem_timedwait (sem=0xa35448 <worker+72>, flags=MONO_SEM_FLAGS_ALERTABLE, timeout_ms=<optimized out>) at ../../mono/utils/mono-coop-semaphore.h:57
#5  worker_park () at threadpool-worker-default.c:391
#6  worker_thread (unused=unused@entry=0x0) at threadpool-worker-default.c:483
#7  0x0000000000628e96 in start_wrapper_internal (stack_ptr=<optimized out>, start_info=0x0) at threads.c:1098
#8  start_wrapper (data=0x7fad54002780) at threads.c:1158
#9  0x00007fada759ce25 in start_thread () from /lib64/libpthread.so.0
#10 0x00007fada70b0bad in clone () from /lib64/libc.so.6

Thread 6 (Thread 0x7fad7ffff700 (LWP 15080)):
#0  0x00007fada75a2afb in do_futex_wait.constprop.1 () from /lib64/libpthread.so.0
#1  0x00007fada75a2b8f in __new_sem_wait_slow.constprop.0 () from /lib64/libpthread.so.0
#2  0x00007fada75a2c2b in sem_wait@@GLIBC_2.2.5 () from /lib64/libpthread.so.0
#3  0x000000000042abfb in mono_os_sem_wait (flags=<optimized out>, sem=<optimized out>) at ../../mono/utils/mono-os-semaphore.h:209
#4  mono_os_sem_timedwait (flags=MONO_SEM_FLAGS_ALERTABLE, timeout_ms=4294967295, sem=0x7fad700085b0) at ../../mono/utils/mono-os-semaphore.h:242
#5  mono_coop_sem_timedwait (sem=0x7fad700085b0, timeout_ms=timeout_ms@entry=4294967295, flags=MONO_SEM_FLAGS_ALERTABLE) at ../../mono/utils/mono-coop-semaphore.h:57
#6  0x000000000067319b in mono_monitor_try_enter_inflated (obj=0x7fada0400718, ms=<optimized out>, allow_interruption=1, id=<optimized out>) at monitor.c:907
#7  0x000000000067478e in mono_monitor_try_enter_internal (allow_interruption=<optimized out>, ms=<optimized out>, obj=<optimized out>) at monitor.c:993
#8  ves_icall_System_Threading_Monitor_Monitor_try_enter_with_atomic_var (obj=<optimized out>, ms=<optimized out>, lockTaken=<optimized out>) at monitor.c:1156
#9  0x0000000041a9bfc7 in ?? ()
#10 0x0000000000000001 in ?? ()
#11 0x00007fada04ec320 in ?? ()
#12 0x00007fad7fffe140 in ?? ()
#13 0x0000000000000000 in ?? ()

Thread 5 (Thread 0x7fad9c870700 (LWP 15168)):
#0  0x00007fada75a2afb in do_futex_wait.constprop.1 () from /lib64/libpthread.so.0
#1  0x00007fada75a2b8f in __new_sem_wait_slow.constprop.0 () from /lib64/libpthread.so.0
#2  0x00007fada75a2c2b in sem_wait@@GLIBC_2.2.5 () from /lib64/libpthread.so.0
#3  0x000000000042abfb in mono_os_sem_wait (flags=<optimized out>, sem=<optimized out>) at ../../mono/utils/mono-os-semaphore.h:209
#4  mono_os_sem_timedwait (flags=MONO_SEM_FLAGS_ALERTABLE, timeout_ms=4294967295, sem=0x7fad700085b0) at ../../mono/utils/mono-os-semaphore.h:242
#5  mono_coop_sem_timedwait (sem=0x7fad700085b0, timeout_ms=timeout_ms@entry=4294967295, flags=MONO_SEM_FLAGS_ALERTABLE) at ../../mono/utils/mono-coop-semaphore.h:57
#6  0x000000000067319b in mono_monitor_try_enter_inflated (obj=0x7fada0400718, ms=<optimized out>, allow_interruption=1, id=<optimized out>) at monitor.c:907
#7  0x000000000067478e in mono_monitor_try_enter_internal (allow_interruption=<optimized out>, ms=<optimized out>, obj=<optimized out>) at monitor.c:993
#8  ves_icall_System_Threading_Monitor_Monitor_try_enter_with_atomic_var (obj=<optimized out>, ms=<optimized out>, lockTaken=<optimized out>) at monitor.c:1156
#9  0x0000000041a9bfc7 in ?? ()
#10 0x0000000000000000 in ?? ()

Thread 4 (Thread 0x7fac773d8700 (LWP 15209)):
#0  0x00007fada75a2afb in do_futex_wait.constprop.1 () from /lib64/libpthread.so.0
#1  0x00007fada75a2b8f in __new_sem_wait_slow.constprop.0 () from /lib64/libpthread.so.0
#2  0x00007fada75a2c2b in sem_wait@@GLIBC_2.2.5 () from /lib64/libpthread.so.0
#3  0x000000000042abfb in mono_os_sem_wait (flags=<optimized out>, sem=<optimized out>) at ../../mono/utils/mono-os-semaphore.h:209
#4  mono_os_sem_timedwait (flags=MONO_SEM_FLAGS_ALERTABLE, timeout_ms=4294967295, sem=0x7fad700085b0) at ../../mono/utils/mono-os-semaphore.h:242
#5  mono_coop_sem_timedwait (sem=0x7fad700085b0, timeout_ms=timeout_ms@entry=4294967295, flags=MONO_SEM_FLAGS_ALERTABLE) at ../../mono/utils/mono-coop-semaphore.h:57
#6  0x000000000067319b in mono_monitor_try_enter_inflated (obj=0x7fada0400718, ms=<optimized out>, allow_interruption=1, id=<optimized out>) at monitor.c:907
#7  0x000000000067478e in mono_monitor_try_enter_internal (allow_interruption=<optimized out>, ms=<optimized out>, obj=<optimized out>) at monitor.c:993
#8  ves_icall_System_Threading_Monitor_Monitor_try_enter_with_atomic_var (obj=<optimized out>, ms=<optimized out>, lockTaken=<optimized out>) at monitor.c:1156
#9  0x0000000041a9bfc7 in ?? ()
#10 0x0000000000000001 in ?? ()
#11 0x00007fada055b5e8 in ?? ()
#12 0x00007fac773d7140 in ?? ()
#13 0x0000000000000000 in ?? ()

Thread 3 (Thread 0x7fad9cc72700 (LWP 15243)):
#0  0x00007fada75a2afb in do_futex_wait.constprop.1 () from /lib64/libpthread.so.0
#1  0x00007fada75a2b8f in __new_sem_wait_slow.constprop.0 () from /lib64/libpthread.so.0
#2  0x00007fada75a2c2b in sem_wait@@GLIBC_2.2.5 () from /lib64/libpthread.so.0
#3  0x000000000042abfb in mono_os_sem_wait (flags=<optimized out>, sem=<optimized out>) at ../../mono/utils/mono-os-semaphore.h:209
#4  mono_os_sem_timedwait (flags=MONO_SEM_FLAGS_ALERTABLE, timeout_ms=4294967295, sem=0x7fad700085b0) at ../../mono/utils/mono-os-semaphore.h:242
#5  mono_coop_sem_timedwait (sem=0x7fad700085b0, timeout_ms=timeout_ms@entry=4294967295, flags=MONO_SEM_FLAGS_ALERTABLE) at ../../mono/utils/mono-coop-semaphore.h:57
#6  0x000000000067319b in mono_monitor_try_enter_inflated (obj=0x7fada0400718, ms=<optimized out>, allow_interruption=1, id=<optimized out>) at monitor.c:907
#7  0x000000000067478e in mono_monitor_try_enter_internal (allow_interruption=<optimized out>, ms=<optimized out>, obj=<optimized out>) at monitor.c:993
#8  ves_icall_System_Threading_Monitor_Monitor_try_enter_with_atomic_var (obj=<optimized out>, ms=<optimized out>, lockTaken=<optimized out>) at monitor.c:1156
#9  0x0000000041a9bfc7 in ?? ()
#10 0x00007fada04c5548 in ?? ()
#11 0x00007fada04c5630 in ?? ()
#12 0x00007fada04c54d8 in ?? ()
#13 0x00007fada04c5598 in ?? ()
#14 0x00007fada04c54d8 in ?? ()
#15 0x00007fad94000960 in ?? ()
#16 0x00007fad9cc71a80 in ?? ()
#17 0x00007fad9cc71670 in ?? ()
#18 0x00007fada04c5630 in ?? ()
#19 0x0000000041a9ae28 in ?? ()
#20 0x00007fada059e380 in ?? ()
#21 0xc0430ccccccccccd in ?? ()
#22 0x00007fada059e380 in ?? ()
#23 0x00007fada059e380 in ?? ()
#24 0xc0330ccccccccccd in ?? ()
#25 0x00007fada059e380 in ?? ()
#26 0x00007fada059e380 in ?? ()
#27 0x0000000000000000 in ?? ()

Thread 2 (Thread 0x7fad7c9ff700 (LWP 15816)):
#0  0x00007fada75a4279 in waitpid () from /lib64/libpthread.so.0
#1  0x00000000004c133c in mono_handle_native_crash (signal=0x71bd46 "SIGABRT", ctx=<optimized out>, info=<optimized out>) at mini-exceptions.c:3048
#2  <signal handler called>
#3  0x00007fada6fe8277 in raise () from /lib64/libc.so.6
#4  0x00007fada6fe9968 in abort () from /lib64/libc.so.6
#5  0x00000000006e1419 in mono_log_write_logfile (log_domain=<optimized out>, level=<optimized out>, hdr=<optimized out>, message=0x7fad48002490 "* Assertion at mini-exceptions.c:2621, condition `gaddr == tls->stack_ovf_guard_base' not met\n") at mono-log-common.c:135
#6  0x00000000006f5d4d in monoeg_g_logv (log_domain=log_domain@entry=0x0, log_level=log_level@entry=G_LOG_LEVEL_ERROR, format=format@entry=0x6ff5a0 "* Assertion at %s:%d, condition `%s' not met\n", args=args@entry=0x7fad7c9fec90) at goutput.c:115
#7  0x00000000006f5f30 in monoeg_assertion_message (format=format@entry=0x6ff5a0 "* Assertion at %s:%d, condition `%s' not met\n") at goutput.c:146
#8  0x00000000004c0dcb in mono_setup_altstack (tls=tls@entry=0x7fad48001b80) at mini-exceptions.c:2621
#9  0x000000000042d2fc in setup_jit_tls_data (stack_start=stack_start@entry=0x7fad7ca00000, abort_func=abort_func@entry=0x42c6d0 <mono_thread_abort>) at mini-runtime.c:931
#10 0x000000000042d357 in setup_jit_tls_data (abort_func=0x42c6d0 <mono_thread_abort>, stack_start=0x7fad7ca00000) at mini-runtime.c:956
#11 mono_thread_start_cb (tid=<optimized out>, stack_start=0x7fad7ca00000, func=<optimized out>) at mini-runtime.c:951
#12 0x0000000000628e15 in start_wrapper_internal (stack_ptr=0x7fad7ca00000, start_info=0x7fad90003dc0) at threads.c:1061
#13 start_wrapper (data=0x7fad90003dc0) at threads.c:1158
#14 0x00007fada759ce25 in start_thread () from /lib64/libpthread.so.0
#15 0x00007fada70b0bad in clone () from /lib64/libc.so.6

Thread 1 (Thread 0x7fada80d3780 (LWP 31039)):
#0  0x00007fada6fe85f2 in sigsuspend () from /lib64/libc.so.6
#1  0x00000000006ee74e in suspend_signal_handler (_dummy=<optimized out>, info=<optimized out>, context=0x7ffe97ecd940) at mono-threads-posix-signals.c:183
#2  <signal handler called>
#3  0x00007fada75a2af9 in do_futex_wait.constprop.1 () from /lib64/libpthread.so.0
#4  0x00007fada75a2b8f in __new_sem_wait_slow.constprop.0 () from /lib64/libpthread.so.0
#5  0x00007fada75a2c2b in sem_wait@@GLIBC_2.2.5 () from /lib64/libpthread.so.0
#6  0x000000000042abfb in mono_os_sem_wait (flags=<optimized out>, sem=<optimized out>) at ../../mono/utils/mono-os-semaphore.h:209
#7  mono_os_sem_timedwait (flags=MONO_SEM_FLAGS_ALERTABLE, timeout_ms=4294967295, sem=0x7fad700085b0) at ../../mono/utils/mono-os-semaphore.h:242
#8  mono_coop_sem_timedwait (sem=0x7fad700085b0, timeout_ms=timeout_ms@entry=4294967295, flags=MONO_SEM_FLAGS_ALERTABLE) at ../../mono/utils/mono-coop-semaphore.h:57
#9  0x000000000067319b in mono_monitor_try_enter_inflated (obj=0x7fada0400718, ms=<optimized out>, allow_interruption=1, id=<optimized out>) at monitor.c:907
#10 0x000000000067478e in mono_monitor_try_enter_internal (allow_interruption=<optimized out>, ms=<optimized out>, obj=<optimized out>) at monitor.c:993
#11 ves_icall_System_Threading_Monitor_Monitor_try_enter_with_atomic_var (obj=<optimized out>, ms=<optimized out>, lockTaken=<optimized out>) at monitor.c:1156
#12 0x0000000041a9bfc7 in ?? ()
#13 0x00007fa9794de2c8 in ?? ()
#14 0x00007ffe97ece150 in ?? ()
#15 0x00007fada0400718 in ?? ()
#16 0x0000000000000000 in ?? ()


Got a SIGABRT while executing native code. This usually indicates
a fatal error in the mono runtime or one of the native libraries 
used by your application.
@benjaminknox benjaminknox changed the title Exception During Run: * Assertion at mini-exceptions.c:2621, condition `gaddr == tls->stack_ovf_guard_base' not met Exception During Runtime: * Assertion at mini-exceptions.c:2621, condition `gaddr == tls->stack_ovf_guard_base' not met Aug 31, 2018
@lewurm lewurm added the area-Runtime label Sep 3, 2018
@lewurm lewurm added this to Bugs Pool in Bugs Week via automation Sep 3, 2018
@lewurm

This comment has been minimized.

Copy link
Member

@lewurm lewurm commented Sep 3, 2018

Thanks for your report! In order to help us to identify the issue, could you:

  1. upload mentioned EXE somewhere? Ideally you strip down the program, but still failing with the mentioned issue. That would help us to track down the program tremendously.

  1. GDB says this:

warning: File "/usr/bin/mono-sgen-gdb.py" auto-loading has been declined by your `auto-load safe-path' set to "$debugdir:$datadir/auto-load:/usr/bin/mono-gdb.py".
To enable execution of this file add
add-auto-load-safe-path /usr/bin/mono-sgen-gdb.py
line to your configuration file "/root/.gdbinit".
To completely disable this security protection add
set auto-load safe-path /
line to your configuration file "/root/.gdbinit".
For more information about this security protection see the
"Auto-loading safe path" section in the GDB manual. E.g., run from the shell:
info "(gdb)Auto-loading safe path"

if you enable this, it should provide better stack traces for managed frames.


Currently, the program runs for a while then crashes giving the above

does it happen everytime you start the program? After what period of time do you see the crash?


  1. I know the /proc/$pid/maps output is very verbose, but the mapping around 0x7fad48001b80 from
#8  0x00000000004c0dcb in mono_setup_altstack (tls=tls@entry=0x7fad48001b80) at mini-exceptions.c:2621

would be kinda interesting. you can use gist.github.com instead and post the link here.

@benjaminknox

This comment has been minimized.

Copy link
Author

@benjaminknox benjaminknox commented Sep 3, 2018

@lewurm Thank you for getting back to me, I cannot upload the EXE because it is closed source and owned by my employer. I will get everything else and post it here.

I have been wondering if the fact that I compiled the EXE on windows with visual studio and simply uploaded it to our CentOS server to run would have anything to do with it?

@benjaminknox

This comment has been minimized.

Copy link
Author

@benjaminknox benjaminknox commented Sep 3, 2018

  1. upload mentioned EXE somewhere? Ideally you strip down the program, but still failing with the mentioned issue. That would help us to track down the program tremendously.

Unfortunately the EXE is proprietary


  1. GDB says this ... add-auto-load-safe-path /usr/bin/mono-sgen-gdb.py ... if you enable this, it should provide better stack traces for managed frames.

I am looking into adding this to our server. This has been added to gdb.


3
Currently, the program runs for a while then crashes giving the above
does it happen everytime you start the program? After what period of time do you see the crash?

This happens every time, no regular interval. I noticed sometimes the program doesn't leave a stack trace. No pattern has yet been found.


  1. I know the /proc/$pid/maps output is very verbose, but the mapping around 0x7fad48001b80 from # 8 0x00000000004c0dcb in mono_setup_altstack (tls=tls@entry=0x7fad48001b80) at mini-exceptions.c:2621 would be kinda interesting. you can use gist.github.com instead and post the link here.

I don't have that particular pid from the stack trace above, and I am also unsure if this is what you wanted, but here is a recent map from that folder: https://gist.github.com/benjaminknox/fd7859c034c54fd4879670ae87898aa4

Let me know if you mean something else.

@vdkhoi

This comment has been minimized.

Copy link

@vdkhoi vdkhoi commented Sep 7, 2018

I also got exact error. However, my application is distributed, which is hard to reproduce. But in my observation, this error happen when I try to copy (or clone) a very large array (2^23 elements) to another array many times, continuously, and very stress.

Mono version: 5.14.0.177
OS: CentOS

@benjaminknox

This comment has been minimized.

Copy link
Author

@benjaminknox benjaminknox commented Sep 10, 2018

@vdkhoi

Yes, I have observed that when we decreased the amount of memory we were using it started happening less.

@lewurm lewurm self-assigned this Oct 1, 2018
@lewurm

This comment has been minimized.

Copy link
Member

@lewurm lewurm commented Oct 1, 2018

potentially related: #10899

we shouldn't assert when mprotect () mono_valloc () fails.

lewurm added a commit to lewurm/mono that referenced this issue Oct 10, 2018
@lewurm lewurm moved this from Bugs Pool to In Progress in Bugs Week Oct 11, 2018
luhenry added a commit that referenced this issue Oct 11, 2018
…w guard page fails (#11064)

* [exceptions] continue program execution when mapping of stack overflow guard page fails

Related: #10415

* turn comment into warning

* fix mono_gc_register_altstack call when no guard page is mapped
@lewurm

This comment has been minimized.

Copy link
Member

@lewurm lewurm commented Oct 12, 2018

@benjaminknox @vdkhoi could you test your application with this patch? #11064

@benjaminknox

This comment has been minimized.

Copy link
Author

@benjaminknox benjaminknox commented Oct 15, 2018

Thank you @lewurm, will let you know

@elrubio

This comment has been minimized.

Copy link

@elrubio elrubio commented Oct 30, 2018

I experience the same or a closely related issue with both, mono v5.16 and the current mono-git v5.21 (compiled just earlier today).

When running certain parts of MaxQuant mass-spectrometry software, mono repeatedly crashes at about the same point in program execution:

// mono 5.16
couldn't allocate guard page, continue without it
couldn't allocate guard page, continue without it
* Assertion at mini-exceptions.c:2953, condition `tls->signal_stack' not met 

or

// mono 5.21
Error: Garbage collector could not allocate 16384 bytes of memory for major heap section.

Our system:
4.18.16-arch1-1-ARCH x86_64
Dual AMD EPYC 7551
256GB RAM

I created a gist with the output of 3 runs with varying versions of mono and MaxQuant. The third file uses the latest mono-git from earlier today.

Observations:

  • The crash happens after a couple of minutes during some single-process highly-multi-threaded number crunching. The process spawns dozens of long-running threads but there is always just one single thread running at a time. It looks like there is some implementation issue concerning suboptimal locking on the MaxQuant side which makes these threads fight for the lock object which ultimately triggers this mono hickup.

  • The input data size affects triggering of the crash: it runs fine with few files (about n<15, total size<10 GB) but reliably crashes when processing more.

  • There is no high pressure neither on system memory nor on CPU during the run: both have just about 50% usage.

Let me know if you need more data. Unfortunately, MaxQuant is closed source and decompiling and dissecting is not an option for me at the moment.

@lewurm

This comment has been minimized.

Copy link
Member

@lewurm lewurm commented Nov 2, 2018

My guess is that mono doesn't handle a out-of-memory situation gracefully enough. @elrubio could you provide precise repro steps?

/cc @BrzVlad

@BrzVlad

This comment has been minimized.

Copy link
Member

@BrzVlad BrzVlad commented Nov 2, 2018

Looks like there is not much to do on the runtime side. The OS refuses to map more memory for the process. If you are confident that the system has more than enough memory available, one common issue that I've encountered on mono + huge memory scenarios is the process exceeding the maximum number of mappings limit. You can tweak that with sudo sysctl vm.max_map_count=1000000.

And yes, if the process runs out of memory, we can crash, instead of throwing a graceful exception.

@elrubio

This comment has been minimized.

Copy link

@elrubio elrubio commented Nov 2, 2018

vm.max_map_count was also my suspect. I raised the value and started a run two days ago. It just finished successfully for the first time, so I can confirm that in this case the number of memory mappings was in fact the culprit.

And yes, if the process runs out of memory, we can crash, instead of throwing a graceful exception.

Well, from a user perspective, a friendly OutOfMemoryException would be incredibly useful for identifiying the root cause of a program crash. In general, an assertion error like this will always look like a bug to me. The error message in the other run was much clearer, it says "heap memory allocation problems". Since I updated mono-git between runs, I don't know if the improved message is the result of a recent code change or because the allocation problem occurred at another point in execution.

Anyway, thanks a lot, @BrzVlad and @lewurm!

@benjaminknox

This comment has been minimized.

Copy link
Author

@benjaminknox benjaminknox commented Nov 8, 2018

We haven't been experiencing this issue anymore, it went away after we reduced the amount of memory our code was using.

@lewurm

This comment has been minimized.

Copy link
Member

@lewurm lewurm commented Nov 8, 2018

I'll close this issue now, if you have any further questions feel free to reopen. Thanks!

@lewurm lewurm closed this Nov 8, 2018
Bugs Week automation moved this from In Progress to Done Nov 8, 2018
@marek-safar marek-safar changed the title Exception During Runtime: * Assertion at mini-exceptions.c:2621, condition `gaddr == tls->stack_ovf_guard_base' not met Assertion when the process runs out of memory Nov 23, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
6 participants
You can’t perform that action at this time.