c# program segfaults when ran under callgrind

[folkertvanheusden]

Description

Hello,

When I try to run a .net 9.0.1 c# program under callgrind (from valgrind), the program segfaults:

==648497== Callgrind, a call-graph generating cache profiler
==648497== Copyright (C) 2002-2017, and GNU GPL'd, by Josef Weidendorfer et al.
==648497== Using Valgrind-3.23.0 and LibVEX; rerun with -h for copyright info
==648497== Command: dotnet run -c Release -- -l /home/folkert/temp/ramdisk/logfile.txt -R roms/GLABIOS.ROM,f000:e000 -F disks/test.img -F disks/xtide.img -X hdds/ide.img -X hdds/ide2.img -p cga,telnet,2300 -p cga,http,8080 -p cga,vnc,5900 -p mda,telnet,2301 -p mda,http,8081 -p mda,vnc,5901 -R roms/ide_xt.bin,d000:0000 -R roms/GLaTICK_0.8.5_AT.ROM,d000:2000 -O midi -O rtc
==648497== 
==648497== For interactive control, run 'callgrind_control -h'.
==648497== 
==648497== Process terminating with default action of signal 11 (SIGSEGV): dumping core
==648497==  Access not within mapped region at address 0x6
==648497==    at 0x50F962A: ??? (in /usr/lib/dotnet/shared/Microsoft.NETCore.App/9.0.1/libcoreclr.so)
==648497==    by 0x52DDEB5: ??? (in /usr/lib/dotnet/shared/Microsoft.NETCore.App/9.0.1/libcoreclr.so)
==648497==    by 0x5E19227F: ??? (in /usr/lib/dotnet/shared/Microsoft.NETCore.App/9.0.1/System.Private.CoreLib.dll)
==648497==    by 0x5E191C61: ??? (in /usr/lib/dotnet/shared/Microsoft.NETCore.App/9.0.1/System.Private.CoreLib.dll)
==648497==    by 0x53AA1FF: ??? (in /usr/lib/dotnet/shared/Microsoft.NETCore.App/9.0.1/libcoreclr.so)
==648497==    by 0x51D8DC8: ??? (in /usr/lib/dotnet/shared/Microsoft.NETCore.App/9.0.1/libcoreclr.so)
==648497==    by 0x52D583C: ??? (in /usr/lib/dotnet/shared/Microsoft.NETCore.App/9.0.1/libcoreclr.so)
==648497==    by 0x55384FB: ??? (in /usr/lib/dotnet/shared/Microsoft.NETCore.App/9.0.1/libcoreclr.so)
==648497==    by 0x553A2FF: ??? (in /usr/lib/dotnet/shared/Microsoft.NETCore.App/9.0.1/libcoreclr.so)
==648497==    by 0x553A009: ??? (in /usr/lib/dotnet/shared/Microsoft.NETCore.App/9.0.1/libcoreclr.so)
==648497==    by 0x556C0C5: ??? (in /usr/lib/dotnet/shared/Microsoft.NETCore.App/9.0.1/libcoreclr.so)
==648497==    by 0x5F6101EF: ??? (in /memfd:doublemapper (deleted))
==648497==  If you believe this happened as a result of a stack
==648497==  overflow in your program's main thread (unlikely but
==648497==  possible), you can try to increase the size of the
==648497==  main thread stack using the --main-stacksize= flag.
==648497==  The main thread stack size used in this run was 16777216.
==648497== 
==648497== Events    : Ir
==648497== Collected : 2090301266
==648497== 
==648497== I   refs:      2,090,301,266
Segmentation fault (core dumped)

This is with this set:

export COMPlus_EnableDiagnostics=0
export DOTNET_GCHeapHardLimit=1C0000000

Reproduction Steps

valgrind --tool=callgrind --main-stacksize=16777216 --fair-sched=yes dotnet run -c Release

Expected behavior

no segfault :-)

Actual behavior

segfault, see bug description

Regression?

don't know

Known Workarounds

No response

Configuration

Microsoft.AspNetCore.App 9.0.1 [/usr/lib/dotnet/shared/Microsoft.AspNetCore.App]
Microsoft.NETCore.App 9.0.1 [/usr/lib/dotnet/shared/Microsoft.NETCore.App]

linux, ubuntu 24.10, x64

Other information

No response

[dotnet-policy-service]

Tagging subscribers to this area: @mangod9
See info in area-owners.md if you want to be subscribed.

[danmoseley]

Can you compare 10.0 preview? If you want to look further - is there a way to resolve the stack trace symbols?

[folkertvanheusden]

Version 10 preview hangs using 100%

I've installed these:

ii  dotnet-runtime-dbg-9.0                          9.0.1-0ubuntu1~24.10.1                   amd64        .NET Runtime debug symbols.
ii  dotnet-sdk-dbg-9.0                              9.0.102-0ubuntu1~24.10.1                 amd64        .NET SDK debug symbols.

but no further strack trace symbols unfortunately.

[mangod9]

Is this a regression in 9? Does it work without the DOTNET_GCHeapHardLimit?

[folkertvanheusden]

Is this a regression in 9? Does it work without the DOTNET_GCHeapHardLimit?

I can't test if it works in 8 as I use Lock from System.Threading.

Regarding running without that environment variable:

GC heap initialization failed with error 0x8007000E
Failed to create CoreCLR, HRESULT: 0x8007000E

[janvorli]

@folkertvanheusden you can get symbols for the libcoreclr.so using the dotnet-symbol tool. You can install it using dotnet tool install -g dotnet-symbol. Then run it on the /usr/lib/dotnet/shared/Microsoft.NETCore.App/9.0.1/libcoreclr.so, it will download the debug symbols side by side with the .so.
I think the packages you have downloaded don't contain native libraries symbols, only the pdbs for the managed files.

[folkertvanheusden]

/usr/lib/dotnet/shared/Microsoft.NETCore.App/9.0.1/libcoreclr.so

Hi,

I'm afraid it says "ERROR: Not Found" on that file. In fact it says that on each file in that directory:

folkert@snsv ~/Projects/DotXT  (master)$ dotnet-symbol /usr/lib/dotnet/shared/Microsoft.NETCore.App/9.0.1/
Downloading from https://msdl.microsoft.com/download/symbols/
ERROR: Not Found: System.Net.WebHeaderCollection.pdb - 'https://msdl.microsoft.com/download/symbols/system.net.webheadercollection.pdb/0366b28c785f985b39c0a22812dc1b2bFFFFFFFF/system.net.webheadercollection.pdb'
ERROR: Not Found: System.Net.WebHeaderCollection.ni.r2rmap - 'https://msdl.microsoft.com/download/symbols/system.net.webheadercollection.ni.r2rmap/r2rmap-v1-dc243efd5ce215feb7ab26d052bf8323/system.net.webheadercollection.ni.r2rmap'
ERROR: Not Found: System.Xml.XmlSerializer.pdb - 'https://msdl.microsoft.com/download/symbols/system.xml.xmlserializer.pdb/ca9b58f3e1613a89dbc9c6df08af50a7FFFFFFFF/system.xml.xmlserializer.pdb'
ERROR: Not Found: System.pdb - 'https://msdl.microsoft.com/download/symbols/system.pdb/8416c95ff00d4ae78aef25abdcb8cb6fFFFFFFFF/system.pdb'
ERROR: Not Found: System.Security.Cryptography.Cng.pdb - 'https://msdl.microsoft.com/download/symbols/system.security.cryptography.cng.pdb/ba92c93150c0447399bc1cfa04b181f4FFFFFFFF/system.security.cryptography.cng.pdb'
...

[folkertvanheusden]

Note that I just got it to work (altough without any debug symbol thus not very useful) by doing:

export COMPlus_EnableDiagnostics=0
export DOTNET_GCHeapHardLimit=1C0000000

export COMPlus_PerfMapEnabled=1
export COMPlus_EnableEventLog=1

valgrind --tool=callgrind --fair-sched=yes bin/Debug/net9.0/dotxt

Thus not a self-contained build.

[janvorli]

Did it say not found for the libcoreclr.so? That should not happen unless the dotnet build is not an official one from Microsoft.

[folkertvanheusden]

Did it say not found for the libcoreclr.so? That should not happen unless the dotnet build is not an official one from Microsoft.

Yes:

$ dotnet-symbol /usr/lib/dotnet/shared/Microsoft.NETCore.App/9.0.1/libcoreclr.so 
Downloading from https://msdl.microsoft.com/download/symbols/
ERROR: Not Found: f80f3206109afecbc0dadf00f6ba2af68470b7.debug - 'https://msdl.microsoft.com/download/symbols/_.debug/elf-buildid-sym-55f80f3206109afecbc0dadf00f6ba2af68470b7/_.debug'

These are microsoft packages:

$ cat /etc/apt/sources.list.d/microsoft-prod.list 
deb [arch=amd64,arm64,armhf signed-by=/usr/share/keyrings/microsoft-prod.gpg] https://packages.microsoft.com/ubuntu/24.04/prod noble main

[janvorli]

Thanks @folkertvanheusden for verifying that.
@tommcdon do you have any idea why dotnet-symbol tool cannot download debug symbols for the libcoreclr.so from the official .NET 9.0.1 build?

[tommcdon]

Thanks @folkertvanheusden for verifying that.
@tommcdon do you have any idea why dotnet-symbol tool cannot download debug symbols for the libcoreclr.so from the official .NET 9.0.1 build?

Sounds like it could be a symbol indexing issue (not uploading to the public symbol server). @hoyosjs

[tommcdon]

Or is this a distro (source build) version of coreclr? Only Microsoft built builds are indexed on to the Microsoft symbol servers.

[folkertvanheusden]

Or is this a distro (source build) version of coreclr? Only Microsoft built builds are indexed on to the Microsoft symbol servers.

No, this package comes from https://packages.microsoft.com/ubuntu/24.04/prod

[folkertvanheusden]

Could it be related to: