CI flakiness: mono interpreter build getting killed

[lewing]

Build Information

Build: https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=998900&view=logs&j=3e93a48f-6568-53f2-76b3-1285c183c39d
Build error leg or test failing: Build / browser-wasm linux Release AllSubsets_Mono_RuntimeTests monointerpreter / Build Tests
Pull request: #114058

Error Message

Fill the error message using step by step known issues guidance.

{
  "ErrorMessage": "",
  "ErrorPattern": "RuntimeVariant=monointerpreter.*exited with code 137",
  "BuildRetry": false,
  "ExcludeConsoleLog": false
}

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=998900
Error message validated: [RuntimeVariant=monointerpreter*exited with code 137.]
Result validation: ❌ Known issue did not match with the provided build.
Validation performed at: 4/1/2025 9:14:37 PM UTC

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=998900
Error message validated: [RuntimeVariant=monointerpreter.*exited with code 137]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 4/1/2025 9:15:42 PM UTC

Report

Build	Definition	Step Name	Console log	Pull Request
1069530	dotnet/runtime	Build Tests	Log	#113956
1069267	dotnet/runtime	Build Tests	Log
1068584	dotnet/runtime	Build Tests	Log
1068493	dotnet/runtime	Build Tests	Log
1068222	dotnet/runtime	Build Tests	Log	#116639
1067996	dotnet/runtime	Build Tests	Log	#113956
1067903	dotnet/runtime	Build Tests	Log	#116626
1067646	dotnet/runtime	Build Tests	Log	#115916
1067636	dotnet/runtime	Build Tests	Log	#116619
1067411	dotnet/runtime	Build Tests	Log	#116626
1066860	dotnet/runtime	Build Tests	Log	#116082
1066684	dotnet/runtime	Build Tests	Log	#116512
1066232	dotnet/runtime	Build Tests	Log	#115393
1066121	dotnet/runtime	Build Tests	Log	#116525
1065389	dotnet/runtime	Build Tests	Log
1065070	dotnet/runtime	Build Tests	Log	#116383
1064009	dotnet/runtime	Build Tests	Log	#116459
1063870	dotnet/runtime	Build Tests	Log
1063280	dotnet/runtime	Build Tests	Log	#116459
1063010	dotnet/runtime	Build Tests	Log
1062955	dotnet/runtime	Build Tests	Log	#115576
1062515	dotnet/runtime	Build Tests	Log	#115537
1062485	dotnet/runtime	Build Tests	Log
1062356	dotnet/runtime	Build Tests	Log	#116428
1062280	dotnet/runtime	Build Tests	Log	#116384
1061978	dotnet/runtime	Build Tests	Log	#116411
1061885	dotnet/runtime	Build Tests	Log
1061710	dotnet/runtime	Build Tests	Log
1061388	dotnet/runtime	Build Tests	Log	#116357
1059782	dotnet/runtime	Build Tests	Log	#116095
1059458	dotnet/runtime	Build Tests	Log	#116326
1059334	dotnet/runtime	Build Tests	Log	#116082
1058553	dotnet/runtime	Build Tests	Log	#114682
1057873	dotnet/runtime	Build Tests	Log
1057689	dotnet/runtime	Build Tests	Log	#115513
1057662	dotnet/runtime	Build Tests	Log	#115774
1057638	dotnet/runtime	Build Tests	Log
1057438	dotnet/runtime	Build Tests	Log	#115855
1057424	dotnet/runtime	Build Tests	Log	#116145
1057101	dotnet/runtime	Build Tests	Log
1056949	dotnet/runtime	Build Tests	Log
1056708	dotnet/runtime	Build Tests	Log	#115513
1055500	dotnet/runtime	Build Tests	Log	#116166
1054323	dotnet/runtime	Build Tests	Log	#116107
1054044	dotnet/runtime	Build Tests	Log	#115609
1053821	dotnet/runtime	Build Tests	Log	#113956
1051675	dotnet/runtime	Build Tests	Log
1050749	dotnet/runtime	Build Tests	Log
1050568	dotnet/runtime	Build Tests	Log	#115704
1049991	dotnet/runtime	Build Tests	Log	#115858
1049202	dotnet/runtime	Build Tests	Log
1047881	dotnet/runtime	Build Tests	Log
1047361	dotnet/runtime	Build Tests	Log	#115818
1045781	dotnet/runtime	Build Tests	Log	#115750
1045198	dotnet/runtime	Build Tests	Log	#115724
1045019	dotnet/runtime	Build Tests	Log	#114148

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
2	18	56

[dotnet-policy-service]

Tagging subscribers to this area: @akoeplinger, @matouskozak
See info in area-owners.md if you want to be subscribed.

[matouskozak]

fyi: @BrzVlad

[pavelsavara]

We use mono interpreter to run msbuild, right ?

This is building 3 groups of many small tests

That combined with WASM toolchain memory demands makes msbuild run out of memory (docker kills it).
This is OOM of the msbuild, not OOM of the individual test projects.
I think this is OOM kill, not timeout because the duration before the kill fluctuates.

As far as I can tell, it never passes for WASM and it's killed in the group 2 already.

Log

2025-05-15T23:09:37.6650337Z   Number of test projects in group 2: 2366
2025-05-15T23:09:37.7341670Z   Building managed test group 2: "/__w/1/s/dotnet.sh" msbuild /__w/1/s/src/tests/build.proj /t:Build "/p:TargetArchitecture=wasm" "/p:Configuration=Release" "/p:LibrariesConfiguration=Release" "/p:TasksConfiguration=Release" "/p:TargetOS=browser" "/p:ToolsOS=" "/p:PackageOS=" "/p:RuntimeFlavor=mono" "/p:RuntimeVariant=monointerpreter" "/p:CLRTestBuildAllTargets=" "/p:UseCodeFlowEnforcement=" "/p:__TestGroupToBuild=2" "/p:__SkipRestorePackages=1" /nodeReuse:false /m:1 /bl:/__w/1/s/artifacts//log/Release/InnerManagedTestBuild.2.binlog /p:ContinuousIntegrationBuild=true "/p:DevTeamProvisioning=-"
2025-05-15T23:09:37.9271786Z   MSBuild version 17.14.0-preview-25177-05+903614e16 for .NET
2025-05-15T23:09:37.9852691Z   
2025-05-15T23:09:40.5433817Z     Number of test projects in group 2: 2366
2025-05-15T23:09:37.7341670Z   Building managed test group 2: "/__w/1/s/dotnet.sh" msbuild /__w/1/s/src/tests/build.proj /t:Build "/p:TargetArchitecture=wasm" "/p:Configuration=Release" "/p:LibrariesConfiguration=Release" "/p:TasksConfiguration=Release" "/p:TargetOS=browser" "/p:ToolsOS=" "/p:PackageOS=" "/p:RuntimeFlavor=mono" "/p:RuntimeVariant=monointerpreter" "/p:CLRTestBuildAllTargets=" "/p:UseCodeFlowEnforcement=" "/p:__TestGroupToBuild=2" "/p:__SkipRestorePackages=1" /nodeReuse:false /m:1 /bl:/__w/1/s/artifacts//log/Release/InnerManagedTestBuild.2.binlog /p:ContinuousIntegrationBuild=true "/p:DevTeamProvisioning=-"
2025-05-15T23:09:37.9271786Z   MSBuild version 17.14.0-preview-25177-05+903614e16 for .NET
2025-05-15T23:09:37.9852691Z   
2025-05-15T23:09:40.5433817Z     Number of test projects in group 2: 2366
2025-05-15T23:09:41.8292504Z     XUnitWrapperGenerator -> /__w/1/s/artifacts/tests/coreclr/browser.wasm.Release/Common/XUnitWrapperGenerator/XUnitWrapperGenerator/XUnitWrapperGenerator.dll
2025-05-15T23:09:42.6596953Z     KeepAliveBoxOpt -> /__w/1/s/artifacts/tests/coreclr/browser.wasm.Release/JIT/Methodical/Boxing/boxunbox/KeepAliveBoxOpt/KeepAliveBoxOpt.dll
2025-05-15T23:09:42.9728086Z     sinlib_cs -> /__w/1/s/artifacts/tests/coreclr/browser.wasm.Release/JIT/Methodical/Boxing/xlang/sinlib_cs/sinlib_cs.dll
2025-05-15T23:09:43.2633097Z     testlib_misc -> /__w/1/s/artifacts/tests/coreclr/browser.wasm.Release/JIT/Methodical/cctor/misc/testlib_misc/testlib_misc.dll
2025-05-15T23:09:43.6004947Z     threads1_cs_do -> /__w/1/s/artifacts/tests/coreclr/browser.wasm.Release/JIT/Methodical/cctor/misc/threads1_cs_do/threads1_cs_do.dll
2025-05-15T23:09:43.8861163Z     testlib_xassem -> /__w/1/s/artifacts/tests/coreclr/browser.wasm.Release/JIT/Methodical/cctor/xassem/testlib_xassem/testlib_xassem.dll
...
2025-05-15T23:23:01.9327210Z     b53977 -> /__w/1/s/artifacts/tests/coreclr/browser.wasm.Release/JIT/Regression/CLR-x86-JIT/V1-M12-Beta2/b53977/b53977/b53977.dll
2025-05-15T23:23:08.5186861Z   /__w/1/s/dotnet.sh: line 27: 50331 Killed                  ${dotnetPath} "$@"
2025-05-15T23:23:08.5814254Z ##[error]src/tests/build.proj(501,5): error MSB3073: The command ""/__w/1/s/dotnet.sh" msbuild /__w/1/s/src/tests/build.proj /t:Build "/p:TargetArchitecture=wasm" "/p:Configuration=Release" "/p:LibrariesConfiguration=Release" "/p:TasksConfiguration=Release" "/p:TargetOS=browser" "/p:ToolsOS=" "/p:PackageOS=" "/p:RuntimeFlavor=mono" "/p:RuntimeVariant=monointerpreter" "/p:CLRTestBuildAllTargets=" "/p:UseCodeFlowEnforcement=" "/p:__TestGroupToBuild=2" "/p:__SkipRestorePackages=1" /nodeReuse:false /m:1 /bl:/__w/1/s/artifacts//log/Release/InnerManagedTestBuild.2.binlog /p:ContinuousIntegrationBuild=true "/p:DevTeamProvisioning=-"" exited with code 137.

[pavelsavara]

I disabled part of the test on browser, because 100% fail rate was not helping anybody. This is ActiveIssue now.