[mono][aot] MonoAOT BDN build crashing with SIGABRT #91819

LoopedBard3 · 2023-09-08T20:10:15Z

Description

The MonoAOT performance runs in the performance pipeline are failing when BDN is trying to build the project with the stacktrace below. Example pipeline run: https://dev.azure.com/dnceng/internal/_build/results?buildId=2262800&view=results, console logs can be found by following any of the failing links in the 'mono AOT micro_mono' failing jobs send to helix step.

cc @SamMonoRT, @radical

Reproduction Steps

Download the payload from the send to helix job onto a local Linux Ubuntu 22.04 x64 machine and run the following with the $(dir) filled in with the full path:

python3 $(dir)/performance/scripts/benchmarks_ci.py --csproj $(dir)/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture x64 -f net8.0 --dotnet-versions 8.0.100-rc.2.23456.6 --bdn-artifacts /home/helixbot/work/BFCE0A6D/w/A9650903/e/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $(dir)/monoaot/mono-aot-cross --customruntimepack $(dir)/monoaot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog  --partition-count 30 --partition-index 0"

Expected behavior

Files are built properly and BDN runs properly.

Actual behavior

Building of the microbenchmarks fails:

[2023/09/08 08:54:13][INFO]   [217/225] System.Private.Xml.dll -> System.Private.Xml.dll.so, System.Private.Xml.dll-llvm.o
[2023/09/08 08:54:13][INFO]   [System.Private.CoreLib.dll] Exec (with response file contents expanded) in /home/helixbot/work/BFCE0A6D/w/A9650903/e/performance/artifacts/bin/MicroBenchmarks/Release/net8.0/Job-CIMUKU/bin/net8.0/linux-x64/publish: MONO_PATH=/home/helixbot/work/BFCE0A6D/w/A9650903/e/performance/artifacts/bin/MicroBenchmarks/Release/net8.0/Job-CIMUKU/bin/net8.0/linux-x64/publish: MONO_ENV_OPTIONS= /home/helixbot/work/BFCE0A6D/p/monoaot/mono-aot-cross --debug --llvm "--aot=mcpu=native,nodebug,llvm-path=/home/helixbot/work/BFCE0A6D/p/monoaot/pack/runtimes/linux-x64/native,outfile=/home/helixbot/work/BFCE0A6D/w/A9650903/e/performance/artifacts/bin/MicroBenchmarks/Release/net8.0/Job-CIMUKU/bin/net8.0/linux-x64/publish/System.Private.CoreLib.dll.so,llvm-outfile=/home/helixbot/work/BFCE0A6D/w/A9650903/e/performance/artifacts/bin/MicroBenchmarks/Release/net8.0/Job-CIMUKU/bin/net8.0/linux-x64/publish/System.Private.CoreLib.dll-llvm.o" "System.Private.CoreLib.dll"
[2023/09/08 08:54:13][INFO] 
[2023/09/08 08:54:13][INFO]   Mono Warning: --llvm not enabled in this runtime.
[2023/09/08 08:54:13][INFO]   Mono Ahead of Time compiler - compiling assembly /home/helixbot/work/BFCE0A6D/w/A9650903/e/performance/artifacts/bin/MicroBenchmarks/Release/net8.0/Job-CIMUKU/bin/net8.0/linux-x64/publish/System.Private.CoreLib.dll
[2023/09/08 08:54:13][INFO]   AOTID F6EE3CE4-B914-B5F2-B8EB-13B84DEF01BF
[2023/09/08 08:54:13][INFO]   Opcode 'xop' missing from machine description file.
[2023/09/08 08:54:13][INFO]   =================================================================
[2023/09/08 08:54:13][INFO]   	Native Crash Reporting
[2023/09/08 08:54:13][INFO]   =================================================================
[2023/09/08 08:54:13][INFO]   Got a SIGABRT while executing native code. This usually indicates
[2023/09/08 08:54:13][INFO]   a fatal error in the mono runtime or one of the native libraries
[2023/09/08 08:54:13][INFO]   used by your application.
[2023/09/08 08:54:13][INFO]   =================================================================
[2023/09/08 08:54:13][INFO]   =================================================================
[2023/09/08 08:54:13][INFO]   	Native stacktrace:
[2023/09/08 08:54:13][INFO]   =================================================================
[2023/09/08 08:54:13][INFO]   	0x5598918ec7f1 - Unknown
[2023/09/08 08:54:13][INFO]   	0x559891893c8e - Unknown
[2023/09/08 08:54:13][INFO]   	0x5598918ec091 - Unknown
[2023/09/08 08:54:13][INFO]   	0x7feca9e42520 - Unknown
[2023/09/08 08:54:13][INFO]   	0x7feca9e96a7c - Unknown
[2023/09/08 08:54:13][INFO]   	0x7feca9e42476 - Unknown
[2023/09/08 08:54:13][INFO]   	0x7feca9e287f3 - Unknown
[2023/09/08 08:54:13][INFO]   	0x5598916b97a5 - Unknown
[2023/09/08 08:54:13][INFO]   	0x5598916d0ab6 - Unknown
[2023/09/08 08:54:13][INFO]   	0x5598916b9b66 - Unknown
[2023/09/08 08:54:13][INFO]   	0x5598916b9c9e - Unknown
[2023/09/08 08:54:13][INFO]   	0x559891889dc7 - Unknown
[2023/09/08 08:54:13][INFO]   	0x5598917fd4c1 - Unknown
[2023/09/08 08:54:13][INFO]   	0x55989180005d - Unknown
[2023/09/08 08:54:13][INFO]   	0x559891872d93 - Unknown
[2023/09/08 08:54:13][INFO]   	0x559891865bef - Unknown
[2023/09/08 08:54:13][INFO]   	0x559891861e52 - Unknown
[2023/09/08 08:54:13][INFO]   	0x55989185f99e - Unknown
[2023/09/08 08:54:13][INFO]   	0x5598917ef01b - Unknown
[2023/09/08 08:54:13][INFO]   	0x5598917eafcb - Unknown
[2023/09/08 08:54:13][INFO]   	0x5598916aa04a - Unknown
[2023/09/08 08:54:13][INFO]   	0x7feca9e29d90 - Unknown
[2023/09/08 08:54:13][INFO]   	0x7feca9e29e40 - Unknown
[2023/09/08 08:54:13][INFO]   	0x5598916a9f19 - Unknown
[2023/09/08 08:54:13][INFO]   =================================================================
[2023/09/08 08:54:13][INFO]   	External Debugger Dump:
[2023/09/08 08:54:13][INFO]   =================================================================
[2023/09/08 08:54:13][INFO]   [New LWP 1018079]
[2023/09/08 08:54:13][INFO]   [Thread debugging using libthread_db enabled]
[2023/09/08 08:54:13][INFO]   Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[2023/09/08 08:54:13][INFO]   0x00007feca9eea49f in __GI___wait4 (pid=1018386, stat_loc=0x7ffc1eec0040, options=0, usage=0x0) at ../sysdeps/unix/sysv/linux/wait4.c:30
[2023/09/08 08:54:13][INFO]   30	../sysdeps/unix/sysv/linux/wait4.c: No such file or directory.
[2023/09/08 08:54:13][INFO]     Id   Target Id                                            Frame
[2023/09/08 08:54:13][INFO]   * 1    Thread 0x7fecaa127740 (LWP 1018074) "mono-aot-cross" 0x00007feca9eea49f in __GI___wait4 (pid=1018386, stat_loc=0x7ffc1eec0040, options=0, usage=0x0) at ../sysdeps/unix/sysv/linux/wait4.c:30
[2023/09/08 08:54:13][INFO]     2    Thread 0x7feca87ff640 (LWP 1018079) "SGen worker"    __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x55989193db18) at ./nptl/futex-internal.c:57
[2023/09/08 08:54:13][INFO]   Thread 2 (Thread 0x7feca87ff640 (LWP 1018079) "SGen worker"):
[2023/09/08 08:54:13][INFO]   #0  __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x55989193db18) at ./nptl/futex-internal.c:57
[2023/09/08 08:54:13][INFO]   #1  __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x55989193db18) at ./nptl/futex-internal.c:87
[2023/09/08 08:54:13][INFO]   #2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x55989193db18, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
[2023/09/08 08:54:13][INFO]   #3  0x00007feca9e93ac1 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x55989193dac8, cond=0x55989193daf0) at ./nptl/pthread_cond_wait.c:503
[2023/09/08 08:54:13][INFO]   #4  ___pthread_cond_wait (cond=0x55989193daf0, mutex=0x55989193dac8) at ./nptl/pthread_cond_wait.c:627
[2023/09/08 08:54:13][INFO]   #5  0x00005598916e5fe3 in ?? ()
[2023/09/08 08:54:13][INFO]   #6  0x00007feca9e94b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
[2023/09/08 08:54:13][INFO]   #7  0x00007feca9f26a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
[2023/09/08 08:54:13][INFO]   Thread 1 (Thread 0x7fecaa127740 (LWP 1018074) "mono-aot-cross"):
[2023/09/08 08:54:13][INFO]   #0  0x00007feca9eea49f in __GI___wait4 (pid=1018386, stat_loc=0x7ffc1eec0040, options=0, usage=0x0) at ../sysdeps/unix/sysv/linux/wait4.c:30
[2023/09/08 08:54:13][INFO]   #1  0x00005598918ec922 in ?? ()
[2023/09/08 08:54:13][INFO]   #2  0x0000559891893c8e in ?? ()
[2023/09/08 08:54:13][INFO]   #3  0x00005598918ec091 in ?? ()
[2023/09/08 08:54:13][INFO]   #4  <signal handler called>
[2023/09/08 08:54:13][INFO]   #5  __pthread_kill_implementation (no_tid=0, signo=6, threadid=140654442346304) at ./nptl/pthread_kill.c:44
[2023/09/08 08:54:13][INFO]   #6  __pthread_kill_internal (signo=6, threadid=140654442346304) at ./nptl/pthread_kill.c:78
[2023/09/08 08:54:13][INFO]   #7  __GI___pthread_kill (threadid=140654442346304, signo=signo@entry=6) at ./nptl/pthread_kill.c:89
[2023/09/08 08:54:13][INFO]   #8  0x00007feca9e42476 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
[2023/09/08 08:54:13][INFO]   #9  0x00007feca9e287f3 in __GI_abort () at ./stdlib/abort.c:79
[2023/09/08 08:54:13][INFO]   #10 0x00005598916b97a5 in ?? ()
[2023/09/08 08:54:13][INFO]   #11 0x00005598916d0ab6 in ?? ()
[2023/09/08 08:54:13][INFO]   #12 0x00005598916b9b66 in ?? ()
[2023/09/08 08:54:13][INFO]   #13 0x00005598916b9c9e in ?? ()
[2023/09/08 08:54:13][INFO]   #14 0x0000559891889dc7 in ?? ()
[2023/09/08 08:54:13][INFO]   #15 0x00005598917fd4c1 in ?? ()
[2023/09/08 08:54:13][INFO]   #16 0x000055989180005d in ?? ()
[2023/09/08 08:54:13][INFO]   #17 0x0000559891872d93 in ?? ()
[2023/09/08 08:54:13][INFO]   #18 0x0000559891865bef in ?? ()
[2023/09/08 08:54:13][INFO]   #19 0x0000559891861e52 in ?? ()
[2023/09/08 08:54:13][INFO]   #20 0x000055989185f99e in ?? ()
[2023/09/08 08:54:13][INFO]   #21 0x00005598917ef01b in ?? ()
[2023/09/08 08:54:13][INFO]   #22 0x00005598917eafcb in ?? ()
[2023/09/08 08:54:13][INFO]   #23 0x00005598916aa04a in ?? ()
[2023/09/08 08:54:13][INFO]   #24 0x00007feca9e29d90 in __libc_start_call_main (main=main@entry=0x5598916aa020, argc=argc@entry=2, argv=argv@entry=0x7ffc1eec2a38) at ../sysdeps/nptl/libc_start_call_main.h:58
[2023/09/08 08:54:13][INFO]   #25 0x00007feca9e29e40 in __libc_start_main_impl (main=0x5598916aa020, argc=2, argv=0x7ffc1eec2a38, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffc1eec2a28) at ../csu/libc-start.c:392
[2023/09/08 08:54:13][INFO]   #26 0x00005598916a9f19 in ?? ()
[2023/09/08 08:54:13][INFO]   [Inferior 1 (process 1018074) detached]
[2023/09/08 08:54:13][INFO]   =================================================================
[2023/09/08 08:54:13][INFO]   	Basic Fault Address Reporting
[2023/09/08 08:54:13][INFO]   =================================================================
[2023/09/08 08:54:13][INFO]   Memory around native instruction pointer (0x7feca9e96a7c):0x7feca9e96a6c  05 00 44 89 e2 89 ee 89 c7 b8 ea 00 00 00 0f 05  ..D.............
[2023/09/08 08:54:13][INFO]   0x7feca9e96a7c  41 89 c5 41 f7 dd 3d 00 f0 ff ff b8 00 00 00 00  A..A..=.........
[2023/09/08 08:54:13][INFO]   0x7feca9e96a8c  44 0f 46 e8 e9 6d ff ff ff 0f 1f 00 48 89 ef e8  D.F..m......H...
[2023/09/08 08:54:13][INFO]   0x7feca9e96a9c  10 a8 ff ff e9 29 ff ff ff 0f 1f 00 48 89 ef e8  .....)......H...
[2023/09/08 08:54:13][INFO] /home/helixbot/work/BFCE0A6D/w/A9650903/e/performance/artifacts/bin/MicroBenchmarks/Release/net8.0/Job-CIMUKU/BenchmarkDotNet.Autogenerated.csproj(74,4): error : Precompiling failed for /home/helixbot/work/BFCE0A6D/w/A9650903/e/performance/artifacts/bin/MicroBenchmarks/Release/net8.0/Job-CIMUKU/bin/net8.0/linux-x64/publish/System.Private.CoreLib.dll
[2023/09/08 08:54:14][INFO]   [218/225] MicroBenchmarks.dll -> MicroBenchmarks.dll.so, MicroBenchmarks.dll-llvm.o
[2023/09/08 08:54:14][INFO]   [219/225] Microsoft.Bcl.AsyncInterfaces.dll -> Microsoft.Bcl.AsyncInterfaces.dll.so, Microsoft.Bcl.AsyncInterfaces.dll-llvm.o
[2023/09/08 08:54:18][INFO]   [220/225] Microsoft.CodeAnalysis.CSharp.dll -> Microsoft.CodeAnalysis.CSharp.dll.so, Microsoft.CodeAnalysis.CSharp.dll-llvm.o
[2023/09/08 08:54:20][INFO]   [221/225] Microsoft.CodeAnalysis.dll -> Microsoft.CodeAnalysis.dll.so, Microsoft.CodeAnalysis.dll-llvm.o
[2023/09/08 08:54:20][INFO]   [222/225] Microsoft.CSharp.dll -> Microsoft.CSharp.dll.so, Microsoft.CSharp.dll-llvm.o
[2023/09/08 08:54:20][INFO]   [223/225] Microsoft.Diagnostics.FastSerialization.dll -> Microsoft.Diagnostics.FastSerialization.dll.so, Microsoft.Diagnostics.FastSerialization.dll-llvm.o
[2023/09/08 08:54:20][INFO]   [224/225] Microsoft.Diagnostics.NETCore.Client.dll -> Microsoft.Diagnostics.NETCore.Client.dll.so, Microsoft.Diagnostics.NETCore.Client.dll-llvm.o
[2023/09/08 08:54:20][INFO] Build FAILED.
[2023/09/08 08:54:20][INFO] CSC : warning CS8002: Referenced assembly 'MicroBenchmarks, Version=42.42.42.42, Culture=neutral, PublicKeyToken=null' does not have a strong name. [/home/helixbot/work/BFCE0A6D/w/A9650903/e/performance/artifacts/bin/MicroBenchmarks/Release/net8.0/Job-CIMUKU/BenchmarkDotNet.Autogenerated.csproj]
[2023/09/08 08:54:20][INFO] /home/helixbot/work/BFCE0A6D/w/A9650903/e/performance/artifacts/bin/MicroBenchmarks/Release/net8.0/Job-CIMUKU/BenchmarkDotNet.Autogenerated.csproj(74,4): error : Precompiling failed for /home/helixbot/work/BFCE0A6D/w/A9650903/e/performance/artifacts/bin/MicroBenchmarks/Release/net8.0/Job-CIMUKU/bin/net8.0/linux-x64/publish/System.Private.CoreLib.dll
[2023/09/08 08:54:20][INFO]     1 Warning(s)
[2023/09/08 08:54:20][INFO]     1 Error(s)
[2023/09/08 08:54:20][INFO] Time Elapsed 00:00:17.91
[2023/09/08 08:54:20][INFO] // command took 18.15 sec and exited with 1

Regression?

This did work in the past. The last successful test used runtime commit: 773c180, and the first known broken test used commit: f3c7893. Here is the github compare: 773c180...f3c7893

Known Workarounds

No response

Configuration

This is running on Linux Ubuntu 22.04 x64.

Other information

No response

The text was updated successfully, but these errors were encountered:

ghost · 2023-09-08T20:14:39Z

Tagging subscribers to this area: @directhex
See info in area-owners.md if you want to be subscribed.

Issue Details

Description

The MonoAOT performance runs in the performance pipeline are failing when BDN is trying to build the project with the stacktrace below. Example pipeline run: https://dev.azure.com/dnceng/internal/_build/results?buildId=2262800&view=results, console logs can be found by following any of the failing links in the 'mono AOT micro_mono' failing jobs send to helix step.

cc @SamMonoRT, @radical

Reproduction Steps

Download the payload from the send to helix job onto a local Linux Ubuntu 22.04 x64 machine and run the following with the $(dir) filled in with the full path:

python3 $(dir)/performance/scripts/benchmarks_ci.py --csproj $(dir)/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture x64 -f net8.0 --dotnet-versions 8.0.100-rc.2.23456.6 --bdn-artifacts /home/helixbot/work/BFCE0A6D/w/A9650903/e/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $(dir)/monoaot/mono-aot-cross --customruntimepack $(dir)/monoaot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog  --partition-count 30 --partition-index 0"

Expected behavior

Files are built properly and BDN runs properly.

Actual behavior

Building of the microbenchmarks fails:

[2023/09/08 08:54:13][INFO]   [217/225] System.Private.Xml.dll -> System.Private.Xml.dll.so, System.Private.Xml.dll-llvm.o
[2023/09/08 08:54:13][INFO]   [System.Private.CoreLib.dll] Exec (with response file contents expanded) in /home/helixbot/work/BFCE0A6D/w/A9650903/e/performance/artifacts/bin/MicroBenchmarks/Release/net8.0/Job-CIMUKU/bin/net8.0/linux-x64/publish: MONO_PATH=/home/helixbot/work/BFCE0A6D/w/A9650903/e/performance/artifacts/bin/MicroBenchmarks/Release/net8.0/Job-CIMUKU/bin/net8.0/linux-x64/publish: MONO_ENV_OPTIONS= /home/helixbot/work/BFCE0A6D/p/monoaot/mono-aot-cross --debug --llvm "--aot=mcpu=native,nodebug,llvm-path=/home/helixbot/work/BFCE0A6D/p/monoaot/pack/runtimes/linux-x64/native,outfile=/home/helixbot/work/BFCE0A6D/w/A9650903/e/performance/artifacts/bin/MicroBenchmarks/Release/net8.0/Job-CIMUKU/bin/net8.0/linux-x64/publish/System.Private.CoreLib.dll.so,llvm-outfile=/home/helixbot/work/BFCE0A6D/w/A9650903/e/performance/artifacts/bin/MicroBenchmarks/Release/net8.0/Job-CIMUKU/bin/net8.0/linux-x64/publish/System.Private.CoreLib.dll-llvm.o" "System.Private.CoreLib.dll"
[2023/09/08 08:54:13][INFO] 
[2023/09/08 08:54:13][INFO]   Mono Warning: --llvm not enabled in this runtime.
[2023/09/08 08:54:13][INFO]   Mono Ahead of Time compiler - compiling assembly /home/helixbot/work/BFCE0A6D/w/A9650903/e/performance/artifacts/bin/MicroBenchmarks/Release/net8.0/Job-CIMUKU/bin/net8.0/linux-x64/publish/System.Private.CoreLib.dll
[2023/09/08 08:54:13][INFO]   AOTID F6EE3CE4-B914-B5F2-B8EB-13B84DEF01BF
[2023/09/08 08:54:13][INFO]   Opcode 'xop' missing from machine description file.
[2023/09/08 08:54:13][INFO]   =================================================================
[2023/09/08 08:54:13][INFO]   	Native Crash Reporting
[2023/09/08 08:54:13][INFO]   =================================================================
[2023/09/08 08:54:13][INFO]   Got a SIGABRT while executing native code. This usually indicates
[2023/09/08 08:54:13][INFO]   a fatal error in the mono runtime or one of the native libraries
[2023/09/08 08:54:13][INFO]   used by your application.
[2023/09/08 08:54:13][INFO]   =================================================================
[2023/09/08 08:54:13][INFO]   =================================================================
[2023/09/08 08:54:13][INFO]   	Native stacktrace:
[2023/09/08 08:54:13][INFO]   =================================================================
[2023/09/08 08:54:13][INFO]   	0x5598918ec7f1 - Unknown
[2023/09/08 08:54:13][INFO]   	0x559891893c8e - Unknown
[2023/09/08 08:54:13][INFO]   	0x5598918ec091 - Unknown
[2023/09/08 08:54:13][INFO]   	0x7feca9e42520 - Unknown
[2023/09/08 08:54:13][INFO]   	0x7feca9e96a7c - Unknown
[2023/09/08 08:54:13][INFO]   	0x7feca9e42476 - Unknown
[2023/09/08 08:54:13][INFO]   	0x7feca9e287f3 - Unknown
[2023/09/08 08:54:13][INFO]   	0x5598916b97a5 - Unknown
[2023/09/08 08:54:13][INFO]   	0x5598916d0ab6 - Unknown
[2023/09/08 08:54:13][INFO]   	0x5598916b9b66 - Unknown
[2023/09/08 08:54:13][INFO]   	0x5598916b9c9e - Unknown
[2023/09/08 08:54:13][INFO]   	0x559891889dc7 - Unknown
[2023/09/08 08:54:13][INFO]   	0x5598917fd4c1 - Unknown
[2023/09/08 08:54:13][INFO]   	0x55989180005d - Unknown
[2023/09/08 08:54:13][INFO]   	0x559891872d93 - Unknown
[2023/09/08 08:54:13][INFO]   	0x559891865bef - Unknown
[2023/09/08 08:54:13][INFO]   	0x559891861e52 - Unknown
[2023/09/08 08:54:13][INFO]   	0x55989185f99e - Unknown
[2023/09/08 08:54:13][INFO]   	0x5598917ef01b - Unknown
[2023/09/08 08:54:13][INFO]   	0x5598917eafcb - Unknown
[2023/09/08 08:54:13][INFO]   	0x5598916aa04a - Unknown
[2023/09/08 08:54:13][INFO]   	0x7feca9e29d90 - Unknown
[2023/09/08 08:54:13][INFO]   	0x7feca9e29e40 - Unknown
[2023/09/08 08:54:13][INFO]   	0x5598916a9f19 - Unknown
[2023/09/08 08:54:13][INFO]   =================================================================
[2023/09/08 08:54:13][INFO]   	External Debugger Dump:
[2023/09/08 08:54:13][INFO]   =================================================================
[2023/09/08 08:54:13][INFO]   [New LWP 1018079]
[2023/09/08 08:54:13][INFO]   [Thread debugging using libthread_db enabled]
[2023/09/08 08:54:13][INFO]   Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[2023/09/08 08:54:13][INFO]   0x00007feca9eea49f in __GI___wait4 (pid=1018386, stat_loc=0x7ffc1eec0040, options=0, usage=0x0) at ../sysdeps/unix/sysv/linux/wait4.c:30
[2023/09/08 08:54:13][INFO]   30	../sysdeps/unix/sysv/linux/wait4.c: No such file or directory.
[2023/09/08 08:54:13][INFO]     Id   Target Id                                            Frame
[2023/09/08 08:54:13][INFO]   * 1    Thread 0x7fecaa127740 (LWP 1018074) "mono-aot-cross" 0x00007feca9eea49f in __GI___wait4 (pid=1018386, stat_loc=0x7ffc1eec0040, options=0, usage=0x0) at ../sysdeps/unix/sysv/linux/wait4.c:30
[2023/09/08 08:54:13][INFO]     2    Thread 0x7feca87ff640 (LWP 1018079) "SGen worker"    __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x55989193db18) at ./nptl/futex-internal.c:57
[2023/09/08 08:54:13][INFO]   Thread 2 (Thread 0x7feca87ff640 (LWP 1018079) "SGen worker"):
[2023/09/08 08:54:13][INFO]   #0  __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x55989193db18) at ./nptl/futex-internal.c:57
[2023/09/08 08:54:13][INFO]   #1  __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x55989193db18) at ./nptl/futex-internal.c:87
[2023/09/08 08:54:13][INFO]   #2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x55989193db18, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
[2023/09/08 08:54:13][INFO]   #3  0x00007feca9e93ac1 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x55989193dac8, cond=0x55989193daf0) at ./nptl/pthread_cond_wait.c:503
[2023/09/08 08:54:13][INFO]   #4  ___pthread_cond_wait (cond=0x55989193daf0, mutex=0x55989193dac8) at ./nptl/pthread_cond_wait.c:627
[2023/09/08 08:54:13][INFO]   #5  0x00005598916e5fe3 in ?? ()
[2023/09/08 08:54:13][INFO]   #6  0x00007feca9e94b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
[2023/09/08 08:54:13][INFO]   #7  0x00007feca9f26a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
[2023/09/08 08:54:13][INFO]   Thread 1 (Thread 0x7fecaa127740 (LWP 1018074) "mono-aot-cross"):
[2023/09/08 08:54:13][INFO]   #0  0x00007feca9eea49f in __GI___wait4 (pid=1018386, stat_loc=0x7ffc1eec0040, options=0, usage=0x0) at ../sysdeps/unix/sysv/linux/wait4.c:30
[2023/09/08 08:54:13][INFO]   #1  0x00005598918ec922 in ?? ()
[2023/09/08 08:54:13][INFO]   #2  0x0000559891893c8e in ?? ()
[2023/09/08 08:54:13][INFO]   #3  0x00005598918ec091 in ?? ()
[2023/09/08 08:54:13][INFO]   #4  <signal handler called>
[2023/09/08 08:54:13][INFO]   #5  __pthread_kill_implementation (no_tid=0, signo=6, threadid=140654442346304) at ./nptl/pthread_kill.c:44
[2023/09/08 08:54:13][INFO]   #6  __pthread_kill_internal (signo=6, threadid=140654442346304) at ./nptl/pthread_kill.c:78
[2023/09/08 08:54:13][INFO]   #7  __GI___pthread_kill (threadid=140654442346304, signo=signo@entry=6) at ./nptl/pthread_kill.c:89
[2023/09/08 08:54:13][INFO]   #8  0x00007feca9e42476 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
[2023/09/08 08:54:13][INFO]   #9  0x00007feca9e287f3 in __GI_abort () at ./stdlib/abort.c:79
[2023/09/08 08:54:13][INFO]   #10 0x00005598916b97a5 in ?? ()
[2023/09/08 08:54:13][INFO]   #11 0x00005598916d0ab6 in ?? ()
[2023/09/08 08:54:13][INFO]   #12 0x00005598916b9b66 in ?? ()
[2023/09/08 08:54:13][INFO]   #13 0x00005598916b9c9e in ?? ()
[2023/09/08 08:54:13][INFO]   #14 0x0000559891889dc7 in ?? ()
[2023/09/08 08:54:13][INFO]   #15 0x00005598917fd4c1 in ?? ()
[2023/09/08 08:54:13][INFO]   #16 0x000055989180005d in ?? ()
[2023/09/08 08:54:13][INFO]   #17 0x0000559891872d93 in ?? ()
[2023/09/08 08:54:13][INFO]   #18 0x0000559891865bef in ?? ()
[2023/09/08 08:54:13][INFO]   #19 0x0000559891861e52 in ?? ()
[2023/09/08 08:54:13][INFO]   #20 0x000055989185f99e in ?? ()
[2023/09/08 08:54:13][INFO]   #21 0x00005598917ef01b in ?? ()
[2023/09/08 08:54:13][INFO]   #22 0x00005598917eafcb in ?? ()
[2023/09/08 08:54:13][INFO]   #23 0x00005598916aa04a in ?? ()
[2023/09/08 08:54:13][INFO]   #24 0x00007feca9e29d90 in __libc_start_call_main (main=main@entry=0x5598916aa020, argc=argc@entry=2, argv=argv@entry=0x7ffc1eec2a38) at ../sysdeps/nptl/libc_start_call_main.h:58
[2023/09/08 08:54:13][INFO]   #25 0x00007feca9e29e40 in __libc_start_main_impl (main=0x5598916aa020, argc=2, argv=0x7ffc1eec2a38, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffc1eec2a28) at ../csu/libc-start.c:392
[2023/09/08 08:54:13][INFO]   #26 0x00005598916a9f19 in ?? ()
[2023/09/08 08:54:13][INFO]   [Inferior 1 (process 1018074) detached]
[2023/09/08 08:54:13][INFO]   =================================================================
[2023/09/08 08:54:13][INFO]   	Basic Fault Address Reporting
[2023/09/08 08:54:13][INFO]   =================================================================
[2023/09/08 08:54:13][INFO]   Memory around native instruction pointer (0x7feca9e96a7c):0x7feca9e96a6c  05 00 44 89 e2 89 ee 89 c7 b8 ea 00 00 00 0f 05  ..D.............
[2023/09/08 08:54:13][INFO]   0x7feca9e96a7c  41 89 c5 41 f7 dd 3d 00 f0 ff ff b8 00 00 00 00  A..A..=.........
[2023/09/08 08:54:13][INFO]   0x7feca9e96a8c  44 0f 46 e8 e9 6d ff ff ff 0f 1f 00 48 89 ef e8  D.F..m......H...
[2023/09/08 08:54:13][INFO]   0x7feca9e96a9c  10 a8 ff ff e9 29 ff ff ff 0f 1f 00 48 89 ef e8  .....)......H...
[2023/09/08 08:54:13][INFO] /home/helixbot/work/BFCE0A6D/w/A9650903/e/performance/artifacts/bin/MicroBenchmarks/Release/net8.0/Job-CIMUKU/BenchmarkDotNet.Autogenerated.csproj(74,4): error : Precompiling failed for /home/helixbot/work/BFCE0A6D/w/A9650903/e/performance/artifacts/bin/MicroBenchmarks/Release/net8.0/Job-CIMUKU/bin/net8.0/linux-x64/publish/System.Private.CoreLib.dll
[2023/09/08 08:54:14][INFO]   [218/225] MicroBenchmarks.dll -> MicroBenchmarks.dll.so, MicroBenchmarks.dll-llvm.o
[2023/09/08 08:54:14][INFO]   [219/225] Microsoft.Bcl.AsyncInterfaces.dll -> Microsoft.Bcl.AsyncInterfaces.dll.so, Microsoft.Bcl.AsyncInterfaces.dll-llvm.o
[2023/09/08 08:54:18][INFO]   [220/225] Microsoft.CodeAnalysis.CSharp.dll -> Microsoft.CodeAnalysis.CSharp.dll.so, Microsoft.CodeAnalysis.CSharp.dll-llvm.o
[2023/09/08 08:54:20][INFO]   [221/225] Microsoft.CodeAnalysis.dll -> Microsoft.CodeAnalysis.dll.so, Microsoft.CodeAnalysis.dll-llvm.o
[2023/09/08 08:54:20][INFO]   [222/225] Microsoft.CSharp.dll -> Microsoft.CSharp.dll.so, Microsoft.CSharp.dll-llvm.o
[2023/09/08 08:54:20][INFO]   [223/225] Microsoft.Diagnostics.FastSerialization.dll -> Microsoft.Diagnostics.FastSerialization.dll.so, Microsoft.Diagnostics.FastSerialization.dll-llvm.o
[2023/09/08 08:54:20][INFO]   [224/225] Microsoft.Diagnostics.NETCore.Client.dll -> Microsoft.Diagnostics.NETCore.Client.dll.so, Microsoft.Diagnostics.NETCore.Client.dll-llvm.o
[2023/09/08 08:54:20][INFO] Build FAILED.
[2023/09/08 08:54:20][INFO] CSC : warning CS8002: Referenced assembly 'MicroBenchmarks, Version=42.42.42.42, Culture=neutral, PublicKeyToken=null' does not have a strong name. [/home/helixbot/work/BFCE0A6D/w/A9650903/e/performance/artifacts/bin/MicroBenchmarks/Release/net8.0/Job-CIMUKU/BenchmarkDotNet.Autogenerated.csproj]
[2023/09/08 08:54:20][INFO] /home/helixbot/work/BFCE0A6D/w/A9650903/e/performance/artifacts/bin/MicroBenchmarks/Release/net8.0/Job-CIMUKU/BenchmarkDotNet.Autogenerated.csproj(74,4): error : Precompiling failed for /home/helixbot/work/BFCE0A6D/w/A9650903/e/performance/artifacts/bin/MicroBenchmarks/Release/net8.0/Job-CIMUKU/bin/net8.0/linux-x64/publish/System.Private.CoreLib.dll
[2023/09/08 08:54:20][INFO]     1 Warning(s)
[2023/09/08 08:54:20][INFO]     1 Error(s)
[2023/09/08 08:54:20][INFO] Time Elapsed 00:00:17.91
[2023/09/08 08:54:20][INFO] // command took 18.15 sec and exited with 1

Regression?

This did work in the past. The last successful test used runtime commit: 773c180, and the first known broken test used commit: f3c7893.

Known Workarounds

No response

Configuration

This is running on Linux Ubuntu 22.04 x64.

Other information

No response

Author:	LoopedBard3
Assignees:	-
Labels:	`tenet-performance-benchmarks`, `untriaged`, `area-Infrastructure-mono`, `needs-area-label`
Milestone:	-

SamMonoRT · 2023-09-08T20:15:49Z

@kotlarmilos @radical - any suggestions?

vargaz · 2023-09-08T20:33:01Z

This message:
Mono Warning: --llvm not enabled in this runtime.
means mono-aot-cross was not compiled with llvm support, which shouldn't happen.

vargaz · 2023-09-08T20:34:16Z

This line:
Opcode 'xop' missing from machine description file.
means that the non-llvm backend received an opcode which is only handled by the llvm backend. This could be caused by either the previous problem, or the simd code mistakenly generating this opcode even in non-llvm mode.

kotlarmilos · 2023-09-08T20:49:11Z

There are no recent changes related to the job setup. There is a simd change a974719 which didn't change the xop opcode.

LoopedBard3 · 2023-09-08T21:03:30Z

I also double checked the performance repo and there are no changes in the timeframe between the runs.

vargaz · 2023-09-09T00:00:54Z

I can repro.

Fixes dotnet#91819.

…91828) Fixes #91819.

dotnet-issue-labeler bot added the needs-area-label An area label is needed to ensure this gets routed to the appropriate area owners label Sep 8, 2023

ghost added the untriaged New issue has not been triaged by the area owner label Sep 8, 2023

SamMonoRT added tenet-performance-benchmarks Issue from performance benchmark area-Infrastructure-mono labels Sep 8, 2023

SamMonoRT added this to the 9.0.0 milestone Sep 8, 2023

ghost removed the untriaged New issue has not been triaged by the area owner label Sep 8, 2023

SamMonoRT added untriaged New issue has not been triaged by the area owner and removed needs-area-label An area label is needed to ensure this gets routed to the appropriate area owners untriaged New issue has not been triaged by the area owner labels Sep 8, 2023

vargaz added a commit to vargaz/runtime that referenced this issue Sep 9, 2023

[mono] Avoid an assert when calling X86Base.Pause in non-llvm mode.

5cdbcdd

Fixes dotnet#91819.

vargaz mentioned this issue Sep 9, 2023

[mono] Avoid an assert when calling X86Base.Pause in non-llvm mode. #91828

Merged

ghost added the in-pr There is an active PR which will close this issue when it is merged label Sep 9, 2023

vargaz closed this as completed in #91828 Sep 9, 2023

vargaz added a commit that referenced this issue Sep 9, 2023

[mono] Avoid an assert when calling X86Base.Pause in non-llvm mode. (#…

64a3c31

…91828) Fixes #91819.

ghost removed the in-pr There is an active PR which will close this issue when it is merged label Sep 9, 2023

matouskozak mentioned this issue Sep 15, 2023

Add MonoAOTLLVM to local script dotnet/performance#3338

Merged

ghost locked as resolved and limited conversation to collaborators Oct 9, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[mono][aot] MonoAOT BDN build crashing with SIGABRT #91819

[mono][aot] MonoAOT BDN build crashing with SIGABRT #91819

LoopedBard3 commented Sep 8, 2023 •

edited

Loading

ghost commented Sep 8, 2023

Description

Reproduction Steps

Expected behavior

Actual behavior

Regression?

Known Workarounds

Configuration

Other information

SamMonoRT commented Sep 8, 2023

vargaz commented Sep 8, 2023

vargaz commented Sep 8, 2023

kotlarmilos commented Sep 8, 2023

LoopedBard3 commented Sep 8, 2023

vargaz commented Sep 9, 2023

[mono][aot] MonoAOT BDN build crashing with SIGABRT #91819

[mono][aot] MonoAOT BDN build crashing with SIGABRT #91819

Comments

LoopedBard3 commented Sep 8, 2023 • edited Loading

Description

Reproduction Steps

Expected behavior

Actual behavior

Regression?

Known Workarounds

Configuration

Other information

ghost commented Sep 8, 2023

Description

Reproduction Steps

Expected behavior

Actual behavior

Regression?

Known Workarounds

Configuration

Other information

SamMonoRT commented Sep 8, 2023

vargaz commented Sep 8, 2023

vargaz commented Sep 8, 2023

kotlarmilos commented Sep 8, 2023

LoopedBard3 commented Sep 8, 2023

vargaz commented Sep 9, 2023

LoopedBard3 commented Sep 8, 2023 •

edited

Loading