Test crashes in arm32 `runtime-coreclr libraries-jitstress-random` pipeline

[BruceForstall]

pipeline: runtime-coreclr libraries-jitstress-random
job: net10.0-linux-Release-arm-jitstress_random_1-(Debian.12.Arm32.Open)Ubuntu.2204.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:debian-12-helix-arm32v7
Tests:

Microsoft.Bcl.Cryptography.Tests Work Item
System.Net.Security.Tests Work Item
System.Net.Security.Unit.Tests Work Item
System.Net.Security.Tests.LoggingTest.EventSource_EventsRaisedAsExpected

https://dev.azure.com/dnceng-public/public/_build/results?buildId=1020494&view=ms.vss-test-web.build-test-results-tab

e.g.,

DOTNET_EnableCrashReport=1
DOTNET_TieredCompilation=0
DOTNET_JitStress=d6
DOTNET_DbgMiniDumpName=/home/helixbot/dotnetbuild/dumps/coredump.%d.dmp
DOTNET_DbgEnableMiniDump=1
...
========================= Begin custom configuration settings ==============================
export __IsXUnitLogCheckerSupported=1
export XUNIT_HIDE_PASSING_OUTPUT_DIAGNOSTICS=1
========================== End custom configuration settings ===============================
----- start Fri Apr 18 05:43:28 PM UTC 2025 =============== To repro directly: =====================================================
pushd .
/root/helix/work/correlation/dotnet exec --runtimeconfig System.Net.Security.Tests.runtimeconfig.json --depsfile System.Net.Security.Tests.deps.json xunit.console.dll System.Net.Security.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing 
popd
===========================================================================================================
/root/helix/work/workitem/e /root/helix/work/workitem/e
  Discovering: System.Net.Security.Tests (method display = ClassAndMethod, method display options = None)
  Discovered:  System.Net.Security.Tests (found 319 of 511 test cases)
  Starting:    System.Net.Security.Tests (parallel test collections = on [2 threads], stop on fail = off)
[createdump] Gathering state for process 65 dotnet
[createdump] Crashing thread 0054 signal 11 (000b)
[createdump] Writing crash report to file /home/helixbot/dotnetbuild/dumps/coredump.65.dmp.crashreport.json
[createdump] Crash report successfully written
[createdump] Writing minidump with heap to file /home/helixbot/dotnetbuild/dumps/coredump.65.dmp
[createdump] Written 280678400 bytes (68525 pages) to core file
[createdump] Target process is alive
[createdump] Dump successfully written in 265ms
waitpid() returned successfully (wstatus 00000000) WEXITSTATUS 0 WTERMSIG 0
./RunTests.sh: line 178:    65 Segmentation fault      (core dumped) "$RUNTIME_PATH/dotnet" exec --runtimeconfig System.Net.Security.Tests.runtimeconfig.json --depsfile System.Net.Security.Tests.deps.json xunit.console.dll System.Net.Security.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing $RSP_FILE
/root/helix/work/workitem/e
----- end Fri Apr 18 05:43:52 PM UTC 2025 ----- exit code 139 ----------------------------------------------------------

Kicked off another run to see if it repros:

https://dev.azure.com/dnceng-public/public/_build/results?buildId=1020635&view=results

[dotnet-policy-service]

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch
See info in area-owners.md if you want to be subscribed.

[BruceForstall]

@dotnet/jit-contrib

[BruceForstall]

In a subsequent test run, arm32 didn't fail but arm64 did, with the same crash failure mode in some of the same tests:

https://dev.azure.com/dnceng-public/public/_build/results?buildId=1020635&view=ms.vss-test-web.build-test-results-tab

This seems to indicate that the stress mode mix, specified by the stress value, does matter.

In the arm32 case, it is:

DOTNET_TieredCompilation=0
DOTNET_JitStress=d6

In the arm64 case, it is:

DOTNET_TieredCompilation=0
DOTNET_JitStress=1e5

[kunalspathak]

Turns out there is AV in compareTypesForCast because castTo == null

    frame #7: 0x0000ffffa17d1808 libcoreclr.so`CEEInfo::compareTypesForCast(CORINFO_CLASS_STRUCT_*, CORINFO_CLASS_STRUCT_*) [inlined] MethodTable::GetFlag(this=0x0000000000000000, flag=enum_flag_Category_Mask) const at methodtable.h:3819:16 [opt]
    frame #8: 0x0000ffffa17d1808 libcoreclr.so`CEEInfo::compareTypesForCast(CORINFO_CLASS_STRUCT_*, CORINFO_CLASS_STRUCT_*) [inlined] MethodTable::IsNullable(this=0x0000000000000000) const at methodtable.h:2784:16 [opt]
    frame #9: 0x0000ffffa17d1808 libcoreclr.so`CEEInfo::compareTypesForCast(CORINFO_CLASS_STRUCT_*, CORINFO_CLASS_STRUCT_*) [inlined] Nullable::IsNullableType(type=TypeHandle @ x8) at object.inl:254:34 [opt]
    frame #10: 0x0000ffffa17d1804 libcoreclr.so`CEEInfo::compareTypesForCast(this=<unavailable>, fromClass=0x0000ffff6441a7c8, toClass=0x0000000000000000) at jitinterface.cpp:4179:14 [opt]
    frame #11: 0x0000ffff5d09f970 libclrjit.so`ValueNumStore::VNForCast(this=0x0000aaaaec52d350, func=VNF_CastClass, castToVN=454, objVN=815) at valuenum.cpp:2715:70 [opt]
    frame #12: 0x0000ffff5d09a444 libclrjit.so`ValueNumStore::VNForFunc(this=0x0000aaaaec52d350, typ=TYP_REF, func=VNF_CastClass, arg0VN=454, arg1VN=815) at valuenum.cpp:2811:25 [opt]

[kunalspathak]

This is happening because we create a class handle node through assertion prop, but never set gtCompileTimeHandle field of it.

runtime/src/coreclr/jit/assertionprop.cpp

Line 3249 in 01e5fa1

newTree = gtNewIconHandleNode(curAssertion->op2.u1.iconVal, curAssertion->op2.GetIconFlag(),

As a result, we end up saving it as 0 in embedded handle map of VNStore during value numbering. When we do optimization in VNForCast, we retrieve that 0 value as castTo and pass it to compareTypesForCast(), where we AV. I spoke with Egor and he have few ideas on fixing it.