Segfault building mono on Linux Kernel 6.2.14

[christopherredden]

On Kernel 6.2.14 (tested on F38) I cannot build mono. I noticed this initially because where I work we have a pre-built Mono from an older kernel version, and this also segfaults on 6.2.14. I was building locally to diagnose the issue, but mono compilation does AOT compilation of framework assemblies, and so the just built mono also segfaults.

I rolled my F38 back to 6.2.9 and everything works fine, including the pre-built Mono we use. To make sure this wasn't a Fedora distro kernel bug, I manually built 6.2.14 and reproduced the same problem.

Steps to Reproduce

1. Install Kernel 6.2.14 (Fedora 38 fully updated includes this)
2. Sync the mono repo here
3. Attempt to build mono, it will fail with a segfault when doing AOT compilation of framework assemblies. In my case, it's not the same one every time, but typically:

AOT [build-linux] mscorlib.dll.so
or
AOT [build-linux] Microsoft.CodeAnalysis.dll.so
or
AOT [build-linux] csc.exe.so
etc..

Every time it is the same location for the segfault (detailed in stacktrace below)

mono_arch_create_specific_trampoline (arg1=0x0, tramp_type=MONO_TRAMPOLINE_VCALL, mem_manager=, code_len=0x7fffa800f0ac) at tramp-amd64.c:610

In this case it's amd specific code, so it is possible this is only reproducible on amd CPUs. I have not tried on an intel machine, unfortunately.

Current Behavior

It segfaults when building.

Expected Behavior

Not segfaulting when building, the same behaviour when using kernel 6.2.9.

On which platforms did you notice this

[ ] macOS
[*] Linux
[ ] Windows

Version Used:
mono/mono commit 3005442 (main as of this post)

Stacktrace

=================================================================
	External Debugger Dump:
=================================================================
[New LWP 26739]

This GDB supports auto-downloading debuginfo from the following URLs:
  <https://debuginfod.fedoraproject.org/>
Enable debuginfod for this session? (y or [n]) [answered N; input not from terminal]
Debuginfod has been disabled.
To make this setting permanent, add 'set debuginfod enabled off' to .gdbinit.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
0x00007f0c85ee7c73 in wait4 () from /lib64/libc.so.6
  Id   Target Id                                       Frame 
* 1    Thread 0x7f0c85e0c040 (LWP 26709) "mono"        0x00007f0c85ee7c73 in wait4 () from /lib64/libc.so.6
  2    Thread 0x7f0c77fff6c0 (LWP 26739) "SGen worker" 0x00007f0c85e961d9 in __futex_abstimed_wait_common () from /lib64/libc.so.6

Thread 2 (Thread 0x7f0c77fff6c0 (LWP 26739) "SGen worker"):
#0  0x00007f0c85e961d9 in __futex_abstimed_wait_common () from /lib64/libc.so.6
#1  0x00007f0c85e98b79 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libc.so.6
#2  0x00000000006e65f1 in mono_os_cond_wait (mutex=0x880d40 <lock>, cond=0x880d00 <work_cond>) at ../../mono/utils/mono-os-mutex.h:219
#3  get_work (job=<synthetic pointer>, do_idle=<synthetic pointer>, work_context=<synthetic pointer>, worker_index=<optimized out>) at sgen-thread-pool.c:167
#4  thread_func (data=<optimized out>) at sgen-thread-pool.c:198
#5  0x00007f0c85e99907 in start_thread () from /lib64/libc.so.6
#6  0x00007f0c85f1f870 in clone3 () from /lib64/libc.so.6

Thread 1 (Thread 0x7f0c85e0c040 (LWP 26709) "mono"):
#0  0x00007f0c85ee7c73 in wait4 () from /lib64/libc.so.6
#1  0x00000000004a6a83 in dump_native_stacktrace (signal=signal@entry=0x7da688 "SIGSEGV", mctx=mctx@entry=0x7ffc0ed0dda0) at mini-posix.c:1076
#2  0x00000000004a6bc9 in mono_dump_native_crash_info (signal=signal@entry=0x7da688 "SIGSEGV", mctx=mctx@entry=0x7ffc0ed0dda0, info=info@entry=0x7ffc0ed0e0b0) at mini-posix.c:1120
#3  0x00000000004572b1 in mono_handle_native_crash (signal=0x7da688 "SIGSEGV", mctx=0x7ffc0ed0dda0, info=0x7ffc0ed0e0b0) at mini-exceptions.c:3470
#4  0x000000000041cabf in mono_sigsegv_signal_handler_debug (_dummy=11, _info=0x7ffc0ed0e0b0, context=0x7ffc0ed0df80, debug_fault_addr=0x0) at mini-runtime.c:3607
#5  <signal handler called>
#6  0x00000000004a27c8 in mono_arch_create_specific_trampoline (arg1=0x0, tramp_type=MONO_TRAMPOLINE_VCALL, mem_manager=<optimized out>, code_len=0x7ffc0ed0e56c) at tramp-amd64.c:610
#7  0x000000000045c6d5 in mono_create_specific_trampoline (arg1=arg1@entry=0x0, tramp_type=tramp_type@entry=MONO_TRAMPOLINE_VCALL, domain=0x903870, code_len=code_len@entry=0x0) at mini-trampolines.c:1339
#8  0x000000000041e4c6 in mini_get_vtable_trampoline (vt=<optimized out>, slot_index=0) at mini-runtime.c:3798
#9  0x000000000061cdbf in mono_class_create_runtime_vtable (error=0x7ffc0ed0e7b0, klass=0x92a6f8, domain=0x903870) at object.c:2239
#10 mono_class_vtable_checked (domain=domain@entry=0x903870, klass=0x92a6f8, error=error@entry=0x7ffc0ed0e7b0) at object.c:1946
#11 0x000000000061feb6 in mono_string_new_size_handle (domain=domain@entry=0x903870, len=len@entry=80, error=error@entry=0x7ffc0ed0e7b0) at object.c:6773
#12 0x000000000061ff7f in mono_string_new_size_checked (domain=domain@entry=0x903870, length=length@entry=80, error=error@entry=0x7ffc0ed0e7b0) at object.c:6788
#13 0x0000000000620993 in mono_string_new_utf16_checked (error=0x7ffc0ed0e7b0, len=80, text=0x975830, domain=0x903870) at object.c:6668
#14 mono_string_new_checked (domain=0x903870, text=text@entry=0x975410 "System.Memory, Version=4.0.1.1, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51", error=error@entry=0x7ffc0ed0e7b0) at object.c:6915
#15 0x000000000064ad39 in mono_string_new_handle (domain=<optimized out>, data=data@entry=0x975410 "System.Memory, Version=4.0.1.1, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51", error=error@entry=0x7ffc0ed0e7b0) at handle.c:384
#16 0x0000000000579826 in mono_try_assembly_resolve (alc=alc@entry=0x0, fname_raw=fname_raw@entry=0x975410 "System.Memory, Version=4.0.1.1, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51", requesting=requesting@entry=0x9376a0, refonly=refonly@entry=0, error=error@entry=0x7ffc0ed0e7b0) at appdomain.c:1334
#17 0x000000000057991e in mono_domain_assembly_postload_search (alc=0x0, requesting=0x9376a0, aname=<optimized out>, refonly=0, postload=<optimized out>, user_data=<optimized out>, error_out=0x7ffc0ed0e850) at appdomain.c:1412
#18 0x000000000057eaad in mono_assembly_invoke_search_hook_internal (alc=0x0, requesting=0x9376a0, aname=aname@entry=0x7ffc0ed0eb40, refonly=0, postload=postload@entry=1) at assembly.c:1985
#19 0x0000000000583a28 in mono_assembly_request_byname (aname=aname@entry=0x7ffc0ed0eb40, req=req@entry=0x7ffc0ed0eac0, status=status@entry=0x7ffc0ed0eb3c) at assembly.c:4701
#20 0x000000000058547b in load_reference_by_aname_default_asmctx (aname=aname@entry=0x7ffc0ed0eb40, assm=assm@entry=0x9376a0, status=status@entry=0x7ffc0ed0eb3c, alc=0x0) at assembly.c:1598
#21 0x0000000000585eac in mono_assembly_load_reference (image=image@entry=0x9098f0, index=index@entry=3) at assembly.c:1771
#22 0x000000000058e74b in mono_class_from_typeref_checked (image=0x9098f0, type_token=16777288, error=0x7ffc0ed0f460) at class.c:206
#23 0x000000000058eaa5 in mono_class_get_checked (image=image@entry=0x9098f0, type_token=16777288, error=error@entry=0x7ffc0ed0f460) at class.c:2874
#24 0x00000000005fd882 in do_mono_metadata_parse_type (error=<optimized out>, rptr=0x7ffc0ed0ee48, ptr=<optimized out>, transient=0, container=0x0, m=0x9098f0, type=0x7ffc0ed0ee70) at metadata.c:4416
#25 mono_metadata_parse_type_internal (m=m@entry=0x9098f0, container=container@entry=0x0, opt_attrs=opt_attrs@entry=0, transient=transient@entry=0, ptr=<optimized out>, rptr=rptr@entry=0x7ffc0ed0ef18, error=<optimized out>) at metadata.c:2188
#26 0x00000000005fda2c in mono_metadata_parse_type_checked (error=<optimized out>, rptr=<optimized out>, ptr=<optimized out>, transient=0, opt_attrs=0, container=0x0, m=0x9098f0) at metadata.c:2242
#27 do_mono_metadata_parse_generic_class (error=<optimized out>, rptr=<optimized out>, ptr=0x7f0c766108b2 "\021\201!\001\003\a\025\022\200\205\001\023", container=0x7ffc0ed0ef20, m=0x9098f0, type=0x7ffc0ed0ef20) at metadata.c:4038
#28 do_mono_metadata_parse_type (error=<optimized out>, rptr=0x7ffc0ed0eef8, ptr=<optimized out>, transient=1, container=0x7ffc0ed0ef20, m=0x9098f0, type=0x7ffc0ed0ef20) at metadata.c:4469
#29 mono_metadata_parse_type_internal (m=m@entry=0x9098f0, container=container@entry=0x0, opt_attrs=opt_attrs@entry=0, transient=transient@entry=1, ptr=<optimized out>, rptr=rptr@entry=0x7ffc0ed0efa8, error=<optimized out>) at metadata.c:2188
#30 0x00000000005fd3a4 in mono_metadata_parse_type_checked (error=0x7ffc0ed0f460, rptr=0x7ffc0ed0efa8, ptr=<optimized out>, transient=1, opt_attrs=0, container=0x0, m=0x9098f0) at metadata.c:2242
#31 mono_metadata_parse_generic_inst (m=m@entry=0x9098f0, container=container@entry=0x0, count=1, ptr=<optimized out>, rptr=rptr@entry=0x7ffc0ed0f038, error=error@entry=0x7ffc0ed0f460) at metadata.c:4005
#32 0x00000000005fda78 in do_mono_metadata_parse_generic_class (error=<optimized out>, rptr=<optimized out>, ptr=0x7f0c766108b1 "\025\021\201!\001\003\a\025\022\200\205\001\023", container=0x0, m=0x9098f0, type=0x7ffc0ed0f040) at metadata.c:4049
#33 do_mono_metadata_parse_type (error=<optimized out>, rptr=0x7ffc0ed0f018, ptr=<optimized out>, transient=1, container=0x0, m=0x9098f0, type=0x7ffc0ed0f040) at metadata.c:4469
#34 mono_metadata_parse_type_internal (m=m@entry=0x9098f0, container=container@entry=0x0, opt_attrs=opt_attrs@entry=0, transient=transient@entry=1, ptr=<optimized out>, rptr=rptr@entry=0x7ffc0ed0f0a8, error=<optimized out>) at metadata.c:2188
#35 0x00000000005ff242 in mono_metadata_parse_type_checked (error=0x7ffc0ed0f460, rptr=0x7ffc0ed0f0a8, ptr=<optimized out>, transient=1, opt_attrs=0, container=0x0, m=0x9098f0) at metadata.c:2242
#36 mono_type_create_from_typespec_checked (image=0x9098f0, type_spec=<optimized out>, error=error@entry=0x7ffc0ed0f460) at metadata.c:6884
#37 0x000000000058eac5 in mono_type_retrieve_from_typespec (error=0x7ffc0ed0f460, did_inflate=<synthetic pointer>, context=0x0, type_spec=452984837, image=0x9098f0) at class.c:2231
#38 mono_class_create_from_typespec (context=0x0, error=0x7ffc0ed0f460, type_spec=452984837, image=0x9098f0) at class.c:2264
#39 mono_class_get_checked (image=image@entry=0x9098f0, type_token=type_token@entry=452984837, error=error@entry=0x7ffc0ed0f460) at class.c:2877
#40 0x000000000058eb9a in mono_class_get_and_inflate_typespec_checked (image=image@entry=0x9098f0, type_token=452984837, context=context@entry=0x0, error=error@entry=0x7ffc0ed0f460) at class.c:2836
#41 0x00000000005f838f in mono_metadata_interfaces_from_typedef_full (meta=meta@entry=0x9098f0, index=index@entry=33554477, interfaces=interfaces@entry=0x7ffc0ed0f218, count=count@entry=0x7ffc0ed0f214, heap_alloc_result=heap_alloc_result@entry=0, context=context@entry=0x0, error=0x7ffc0ed0f460) at metadata.c:5316
#42 0x00000000005938e9 in mono_class_create_from_typedef (image=0x9098f0, type_token=33554477, error=0x7ffc0ed0f460) at class-init.c:586
#43 0x000000000058eab5 in mono_class_get_checked (image=image@entry=0x9098f0, type_token=33554477, error=error@entry=0x7ffc0ed0f460) at class.c:2871
#44 0x00000000005e0534 in mono_get_method_from_token (error=0x7ffc0ed0f460, used_context=0x7ffc0ed0f314, context=0x0, klass=0x0, token=100663513, image=0x9098f0) at loader.c:1088
#45 mono_get_method_checked (image=0x9098f0, token=token@entry=100663513, klass=klass@entry=0x0, context=context@entry=0x0, error=error@entry=0x7ffc0ed0f460) at loader.c:1219
#46 0x000000000044616b in collect_methods (acfg=0x939f10) at aot-compiler.c:12189
#47 mono_compile_assembly (ass=ass@entry=0x9376a0, opts=opts@entry=374434303, aot_options=aot_options@entry=0x8f24a0 "bind-to-runtime-version,outfile=./../../class/lib/build-linux//Microsoft.CodeAnalysis.dll.so", global_aot_state=global_aot_state@entry=0x7ffc0ed0f630) at aot-compiler.c:14012
#48 0x000000000042cd3e in main_thread_handler (user_data=<synthetic pointer>) at driver.c:1445
#49 mono_main (argc=<optimized out>, argv=<optimized out>) at driver.c:2744
#50 0x000000000041c29f in mono_main_with_options (argv=<optimized out>, argc=<optimized out>) at /home/christopherredden/Devel/mono/mono/mini/main.c:54
#51 main (argc=<optimized out>, argv=<optimized out>) at /home/christopherredden/Devel/mono/mono/mini/main.c:402```

[zboszor]

I ran into the same issue, see the meta-mono issue and the Fedora issue

The crash problems started with 6.2.13, and it's also present with 6.3.1 from https://koji.fedoraproject.org/koji/buildinfo?buildID=2196477

[raphaelr]

We're also getting segfaults on recent kernels (at least 6.1.26 and 6.3.0 are affected). But only when running mono on certain executables: NixOS/nixpkgs#229476

There's a kernel regression where mmap(2) with flag MAP_32BIT is broken (returns ENOMEM even if enough virtual memory space in the first 4GB is available) (ref)

If it is the same issue, try applying the following patch and see if it helps:

Patch

From e019d4ac4735e774d0a3c15bb36d5eb3ebfa3053 Mon Sep 17 00:00:00 2001
From: Raphael Robatsch <raphael-git@tapesoftware.net>
Date: Tue, 9 May 2023 11:55:02 +0200
Subject: [PATCH] Disable mmap(MAP_32BIT) support

mmap(2) with flag MAP_32BIT can erroneously return ENOMEM on recent
kernels. Disable MAP_32BIT support for now.

Reference: https://lore.kernel.org/linux-mm/cb8dc31a-fef2-1d09-f133-e9f7b9f9e77a@sony.com/
Reference: https://lore.kernel.org/all/20230414185919.4175572-1-Liam.Howlett@oracle.com/T/#m00a0ac8a72bf2f26711b7f8cc56612a8ef62c3d0
---
 mono/mini/mini-amd64.h    | 2 --
 mono/utils/mono-codeman.c | 4 ----
 2 files changed, 6 deletions(-)

diff --git a/mono/mini/mini-amd64.h b/mono/mini/mini-amd64.h
index b321743b67d..0a81bb4bd6b 100644
--- a/mono/mini/mini-amd64.h
+++ b/mono/mini/mini-amd64.h
@@ -390,9 +390,7 @@ typedef struct {

 #endif /* !HOST_WIN32 */

-#if !defined(__linux__) && !defined(__sun)
 #define MONO_ARCH_NOMAP32BIT 1
-#endif

 #ifdef TARGET_WIN32
 #define MONO_AMD64_ARG_REG1 AMD64_RCX
diff --git a/mono/utils/mono-codeman.c b/mono/utils/mono-codeman.c
index 234aac4b0ca..5eccda92bd0 100644
--- a/mono/utils/mono-codeman.c
+++ b/mono/utils/mono-codeman.c
@@ -68,11 +68,7 @@ static const MonoCodeManagerCallbacks *code_manager_callbacks;
 #define MAX_WASTAGE 32
 #define MIN_BSIZE 32

-#ifdef __x86_64__
-#define ARCH_MAP_FLAGS MONO_MMAP_32BIT
-#else
 #define ARCH_MAP_FLAGS 0
-#endif

 #define MONO_PROT_RWX (MONO_MMAP_READ|MONO_MMAP_WRITE|MONO_MMAP_EXEC|MONO_MMAP_JIT)

--
2.40.1

[zboszor]

We're also getting segfaults on recent kernels (at least 6.1.26 and 6.3.0 are affected). But only when running mono on certain executables: NixOS/nixpkgs#229476

There's a kernel regression where mmap(2) with flag MAP_32BIT is broken (returns ENOMEM even if enough virtual memory space in the first 4GB is available) (ref)

If it is the same issue, try applying the following patch and see if it helps:

It was indeed the same problem. Building mono-native succeeds on Fedora 38 with kernel 6.3.1

[akien-mga]

Took me a while to parse the Linux kernel mailing list referenced in @raphaelr's patch, so I figured I'd link it here:

The regression has been acknowledged upstream and a fix has been merged in the mm-unstable branch: https://lore.kernel.org/linux-mm/20230505145829.74574-1-zhangpeng.00@bytedance.com/

There's no indication on when this will make it to mainline kernel and whether the fix will be backported to the affected 6.1, 6.2, and 6.3 branches.

If/when it is, then I assume the workaround to disable mmap support in Mono would no longer be necessary.

[raphaelr]

Should be fixed in 6.1.30 and 6.3.4, (see)

[christopherredden]

I can confirm Kernel 6.3.4 fixed it for me, so I'm going to close this. Thanks to everyone that contributed here :)