Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

System.Net.Requests.Tests failing with a crash #55714

Closed
runfoapp bot opened this issue Jul 15, 2021 · 7 comments
Closed

System.Net.Requests.Tests failing with a crash #55714

runfoapp bot opened this issue Jul 15, 2021 · 7 comments
Labels
area-System.Net
Milestone
6.0.0

Comments

@runfoapp
Copy link

runfoapp bot commented Jul 15, 2021 •
edited
Loading

Runfo Tracking Issue: System.Net.Requests.Tests failing with a crash

Build Definition Kind Run Name Console Core Dump Test Results Run Client
1238781 runtime Rolling net6.0-Linux-Release-x64-mono_interpreter_release-Debian.9.Amd64.Open console.log runclient.py
1238781 runtime Rolling net6.0-Linux-Release-arm-CoreCLR_checked-(Ubuntu.1804.Arm32.Open)Ubuntu.1804.Armarch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7-bfcd90a-20200121150440 console.log runclient.py
1238781 runtime Rolling net6.0-Linux-Release-arm64-CoreCLR_checked-(Alpine.312.Arm64.Open)Ubuntu.1804.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:alpine-3.12-helix-arm64v8-20200602002604-25f8a3e console.log runclient.py
1238781 runtime Rolling net6.0-Linux-Release-arm64-CoreCLR_release-(Ubuntu.1804.ArmArch.Open)Ubuntu.1804.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-16.04-helix-arm64v8-20210106155927-56c6673 console.log runclient.py
1236633 runtime PR 55613 net6.0-Linux-Debug-x64-Mono_release-(Fedora.32.Amd64.Open)ubuntu.1604.amd64.open@mcr.microsoft.com/dotnet-buildtools/prereqs:fedora-32-helix-20210710031120-870c408 console.log core dump runclient.py
1236304 runtime PR 54580 net6.0-windows-Debug-x64-CoreCLR_checked-Windows.10.Amd64.Open console.log core dump runclient.py
1236304 runtime PR 54580 net6.0-Linux-Debug-x64-CoreCLR_checked-Ubuntu.1804.Amd64.Open console.log core dump runclient.py
1236278 runtime PR 55607 net6.0-Linux-Debug-x64-Mono_release-(Fedora.32.Amd64.Open)ubuntu.1604.amd64.open@mcr.microsoft.com/dotnet-buildtools/prereqs:fedora-32-helix-20210710031120-870c408 console.log core dump runclient.py
1236197 runtime PR 55531 net6.0-Linux-Debug-x64-Mono_release-(Fedora.32.Amd64.Open)ubuntu.1604.amd64.open@mcr.microsoft.com/dotnet-buildtools/prereqs:fedora-32-helix-20210710031120-870c408 console.log core dump runclient.py
1236179 runtime Rolling net6.0-Linux-Release-x64-Mono_release-SLES.12.Amd64.Open console.log runclient.py
1236179 runtime Rolling net6.0-Linux-Release-x64-CoreCLR_release-SLES.12.Amd64.Open console.log runclient.py
1236005 runtime PR 55384 net6.0-Linux-Debug-x64-Mono_release-(Fedora.32.Amd64.Open)ubuntu.1604.amd64.open@mcr.microsoft.com/dotnet-buildtools/prereqs:fedora-32-helix-20210710031120-870c408 console.log core dump runclient.py
1235600 runtime PR 55592 net6.0-Linux-Debug-x64-Mono_release-(Fedora.32.Amd64.Open)ubuntu.1604.amd64.open@mcr.microsoft.com/dotnet-buildtools/prereqs:fedora-32-helix-20210710031120-870c408 console.log core dump runclient.py
1235397 runtime PR 55586 net6.0-Linux-Debug-x64-Mono_release-(Fedora.32.Amd64.Open)ubuntu.1604.amd64.open@mcr.microsoft.com/dotnet-buildtools/prereqs:fedora-32-helix-20210710031120-870c408 console.log core dump runclient.py
1235376 runtime PR 55564 net6.0-Linux-Debug-x64-Mono_release-(Fedora.32.Amd64.Open)ubuntu.1604.amd64.open@mcr.microsoft.com/dotnet-buildtools/prereqs:fedora-32-helix-20210710031120-870c408 console.log core dump runclient.py
1235349 runtime Rolling net6.0-Linux-Release-x64-Mono_release-(Fedora.32.Amd64.Open)ubuntu.1604.amd64.open@mcr.microsoft.com/dotnet-buildtools/prereqs:fedora-32-helix-20210710031120-870c408 console.log core dump runclient.py
1235034 runtime PR 55565 net6.0-Linux-Debug-x64-Mono_release-(Fedora.32.Amd64.Open)ubuntu.1604.amd64.open@mcr.microsoft.com/dotnet-buildtools/prereqs:fedora-32-helix-20210710031120-870c408 console.log core dump runclient.py
1234989 runtime PR 55430 net6.0-Linux-Debug-x64-Mono_release-(Fedora.32.Amd64.Open)ubuntu.1604.amd64.open@mcr.microsoft.com/dotnet-buildtools/prereqs:fedora-32-helix-20210710031120-870c408 console.log core dump runclient.py
1234982 runtime PR 55327 net6.0-Linux-Debug-x64-Mono_release-(Fedora.32.Amd64.Open)ubuntu.1604.amd64.open@mcr.microsoft.com/dotnet-buildtools/prereqs:fedora-32-helix-20210710031120-870c408 console.log core dump runclient.py
1234728 runtime PR 55258 net6.0-Linux-Release-arm-CoreCLR_checked-(Ubuntu.1804.Arm32.Open)Ubuntu.1804.Armarch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7-bfcd90a-20200121150440 console.log core dump runclient.py
1234594 runtime Rolling net6.0-Linux-Release-x64-Mono_release-(Fedora.32.Amd64.Open)ubuntu.1604.amd64.open@mcr.microsoft.com/dotnet-buildtools/prereqs:fedora-32-helix-20210710031120-870c408 console.log core dump runclient.py
1234580 runtime PR 55398 net6.0-Linux-Debug-x64-Mono_release-(Fedora.32.Amd64.Open)ubuntu.1604.amd64.open@mcr.microsoft.com/dotnet-buildtools/prereqs:fedora-32-helix-20210710031120-870c408 console.log core dump runclient.py
1234437 runtime Rolling net6.0-Linux-Release-x64-Mono_release-(Fedora.32.Amd64.Open)ubuntu.1604.amd64.open@mcr.microsoft.com/dotnet-buildtools/prereqs:fedora-32-helix-20210710031120-870c408 console.log core dump runclient.py
1234093 runtime PR 55425 net6.0-Linux-Debug-x64-Mono_release-(Fedora.32.Amd64.Open)ubuntu.1604.amd64.open@mcr.microsoft.com/dotnet-buildtools/prereqs:fedora-32-helix-20210710031120-870c408 console.log core dump runclient.py
1234093 runtime PR 55425 net6.0-Linux-Debug-x64-Mono_release-(Fedora.32.Amd64.Open)ubuntu.1604.amd64.open@mcr.microsoft.com/dotnet-buildtools/prereqs:fedora-32-helix-20210710031120-870c408 console.log core dump runclient.py
1234041 runtime PR 55499 net6.0-Linux-Release-arm64-CoreCLR_checked-(Alpine.312.Arm64.Open)Ubuntu.1804.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:alpine-3.12-helix-arm64v8-20200602002604-25f8a3e console.log core dump runclient.py
1234024 runtime PR 55458 net6.0-Linux-Debug-x64-Mono_release-(Fedora.32.Amd64.Open)ubuntu.1604.amd64.open@mcr.microsoft.com/dotnet-buildtools/prereqs:fedora-32-helix-20210710031120-870c408 console.log core dump runclient.py
1233610 runtime PR 55533 net6.0-Linux-Debug-x64-Mono_release-(Fedora.32.Amd64.Open)ubuntu.1604.amd64.open@mcr.microsoft.com/dotnet-buildtools/prereqs:fedora-32-helix-20210710031120-870c408 console.log core dump runclient.py
1233354 runtime PR 55480 net6.0-Linux-Debug-x64-CoreCLR_release-SLES.15.Amd64.Open console.log core dump runclient.py
1233170 runtime PR 55369 net6.0-Linux-Debug-x64-CoreCLR_release-(Centos.8.Amd64.Open)Ubuntu.1604.Amd64.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:centos-8-helix-20201229003624-c1bf759 console.log core dump runclient.py
1233170 runtime PR 55369 net6.0-Linux-Debug-x64-CoreCLR_release-Ubuntu.1804.Amd64.Open console.log core dump runclient.py
1233089 runtime PR 55395 net6.0-Linux-Debug-x64-Mono_release-(Fedora.32.Amd64.Open)ubuntu.1604.amd64.open@mcr.microsoft.com/dotnet-buildtools/prereqs:fedora-32-helix-20210710031120-870c408 console.log core dump runclient.py
1233024 runtime PR 55520 net6.0-Linux-Debug-x64-CoreCLR_release-(Debian.10.Amd64.Open)ubuntu.1604.amd64.open@mcr.microsoft.com/dotnet-buildtools/prereqs:debian-10-helix-amd64-bfcd90a-20200121150006 console.log core dump runclient.py
1232915 runtime PR 55430 net6.0-Linux-Debug-x64-CoreCLR_release-Ubuntu.1804.Amd64.Open console.log core dump runclient.py
1232861 runtime Rolling net6.0-Linux-Release-x64-CoreCLR_checked-Ubuntu.1804.Amd64.Open console.log core dump runclient.py
1232822 runtime PR 55338 net6.0-Linux-Debug-x64-Mono_release-(Fedora.32.Amd64.Open)ubuntu.1604.amd64.open@mcr.microsoft.com/dotnet-buildtools/prereqs:fedora-32-helix-20210710031120-870c408 console.log core dump runclient.py
1232777 runtime PR 55513 net6.0-Linux-Debug-x64-Mono_release-(Fedora.32.Amd64.Open)ubuntu.1604.amd64.open@mcr.microsoft.com/dotnet-buildtools/prereqs:fedora-32-helix-20210710031120-870c408 console.log core dump runclient.py
1232777 runtime PR 55513 net6.0-Linux-Debug-x64-CoreCLR_release-Ubuntu.1804.Amd64.Open console.log core dump runclient.py
1232718 runtime PR 55395 net6.0-Linux-Debug-x64-Mono_release-(Fedora.32.Amd64.Open)ubuntu.1604.amd64.open@mcr.microsoft.com/dotnet-buildtools/prereqs:fedora-32-helix-20210710031120-870c408 console.log core dump runclient.py
1232607 runtime PR 55504 net6.0-Linux-Debug-x64-Mono_release-(Fedora.32.Amd64.Open)ubuntu.1604.amd64.open@mcr.microsoft.com/dotnet-buildtools/prereqs:fedora-32-helix-20210710031120-870c408 console.log core dump runclient.py
1232607 runtime PR 55504 net6.0-Linux-Release-arm-CoreCLR_checked-(Ubuntu.1804.Arm32.Open)Ubuntu.1804.Armarch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7-bfcd90a-20200121150440 console.log core dump runclient.py
1232415 runtime PR 55500 net6.0-Linux-Debug-x64-Mono_release-(Fedora.32.Amd64.Open)ubuntu.1604.amd64.open@mcr.microsoft.com/dotnet-buildtools/prereqs:fedora-32-helix-20210710031120-870c408 console.log core dump runclient.py
1232399 runtime PR 55499 net6.0-Linux-Debug-x64-Mono_release-(Fedora.32.Amd64.Open)ubuntu.1604.amd64.open@mcr.microsoft.com/dotnet-buildtools/prereqs:fedora-32-helix-20210710031120-870c408 console.log core dump runclient.py
1232381 runtime PR 55460 net6.0-Linux-Debug-x64-CoreCLR_checked-Ubuntu.1804.Amd64.Open console.log core dump runclient.py
1232366 runtime PR 55478 net6.0-Linux-Debug-x64-CoreCLR_release-(Fedora.32.Amd64.Open)ubuntu.1604.amd64.open@mcr.microsoft.com/dotnet-buildtools/prereqs:fedora-32-helix-20210710031120-870c408 console.log core dump runclient.py
1232365 runtime Rolling net6.0-Linux-Release-x64-CoreCLR_checked-Ubuntu.1804.Amd64.Open console.log core dump runclient.py
1232171 runtime PR 55460 net6.0-Linux-Debug-x64-CoreCLR_release-SLES.15.Amd64.Open console.log core dump runclient.py
1232164 runtime PR 51653 net6.0-Linux-Release-arm-CoreCLR_checked-(Ubuntu.1804.Arm32.Open)Ubuntu.1804.Armarch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7-bfcd90a-20200121150440 console.log core dump runclient.py
1232111 runtime Rolling net6.0-Linux-Release-x64-CoreCLR_checked-Ubuntu.1804.Amd64.Open console.log core dump runclient.py
1231886 runtime PR 55467 net6.0-OSX-Debug-x64-CoreCLR_release-OSX.1015.Amd64.Open console.log runclient.py
1231886 runtime PR 55467 net6.0-OSX-Debug-x64-CoreCLR_release-OSX.1014.Amd64.Open console.log core dump runclient.py
1231886 runtime PR 55467 net6.0-OSX-Debug-x64-CoreCLR_checked-OSX.1013.Amd64.Open console.log core dump runclient.py
1231886 runtime PR 55467 net6.0-windows-Debug-x64-CoreCLR_release-(Windows.Nano.1809.Amd64.Open)windows.10.amd64.serverrs5.open@mcr.microsoft.com/dotnet-buildtools/prereqs:nanoserver-1809-helix-amd64-08e8e40-20200107182504 console.log runclient.py
1231886 runtime PR 55467 net6.0-windows-Debug-x64-CoreCLR_release-Windows.81.Amd64.Open console.log core dump runclient.py
1231886 runtime PR 55467 net6.0-windows-Debug-x64-CoreCLR_checked-Windows.10.Amd64.Open console.log core dump runclient.py
1231434 runtime PR 55467 net6.0-OSX-Debug-x64-CoreCLR_checked-OSX.1013.Amd64.Open console.log runclient.py
1231434 runtime PR 55467 net6.0-windows-Debug-x64-CoreCLR_checked-Windows.10.Amd64.Open console.log core dump runclient.py
1231434 runtime PR 55467 net6.0-Linux-Release-arm64-CoreCLR_checked-(Alpine.312.Arm64.Open)Ubuntu.1804.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:alpine-3.12-helix-arm64v8-20200602002604-25f8a3e console.log core dump runclient.py
1231434 runtime PR 55467 net6.0-Linux-Debug-x64-CoreCLR_checked-(Alpine.312.Amd64.Open)ubuntu.1604.amd64.open@mcr.microsoft.com/dotnet-buildtools/prereqs:alpine-3.12-helix-20200602002622-e06dc59 console.log core dump runclient.py
1231434 runtime PR 55467 net6.0-Linux-Debug-x64-CoreCLR_checked-Ubuntu.1804.Amd64.Open console.log core dump runclient.py
1230534 runtime PR 54954 net6.0-Linux-Release-arm-CoreCLR_checked-(Ubuntu.1804.Arm32.Open)Ubuntu.1804.Armarch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7-bfcd90a-20200121150440 console.log core dump runclient.py
1228008 runtime PR 54954 net6.0-windows-Release-x86-CoreCLR_checked-Windows.10.Amd64.Open console.log core dump runclient.py
1228008 runtime PR 54954 net6.0-Linux-Release-arm64-CoreCLR_checked-(Alpine.312.Arm64.Open)Ubuntu.1804.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:alpine-3.12-helix-arm64v8-20200602002604-25f8a3e console.log core dump runclient.py
1228008 runtime PR 54954 net6.0-Linux-Release-arm-CoreCLR_checked-(Alpine.313.Arm32.Open)Ubuntu.1804.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:alpine-3.13-helix-arm32v7-20210414141857-1ea6b0a console.log core dump runclient.py
1228008 runtime PR 54954 net6.0-Linux-Release-arm-CoreCLR_checked-(Ubuntu.1804.Arm32.Open)Ubuntu.1804.Armarch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7-bfcd90a-20200121150440 console.log core dump runclient.py
1228008 runtime PR 54954 net6.0-Linux-Debug-x64-CoreCLR_checked-(Alpine.312.Amd64.Open)ubuntu.1604.amd64.open@mcr.microsoft.com/dotnet-buildtools/prereqs:alpine-3.12-helix-20200602002622-e06dc59 console.log core dump runclient.py
1228008 runtime PR 54954 net6.0-OSX-Debug-x64-CoreCLR_checked-OSX.1013.Amd64.Open console.log core dump runclient.py
1227837 runtime PR 55372 net6.0-windows-Release-x86-CoreCLR_checked-Windows.10.Amd64.Open console.log core dump runclient.py
1227837 runtime PR 55372 net6.0-windows-Debug-x64-CoreCLR_checked-Windows.10.Amd64.Open console.log core dump runclient.py
1227837 runtime PR 55372 net6.0-Linux-Release-arm64-CoreCLR_checked-(Alpine.312.Arm64.Open)Ubuntu.1804.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:alpine-3.12-helix-arm64v8-20200602002604-25f8a3e console.log core dump runclient.py
1227837 runtime PR 55372 net6.0-Linux-Release-arm-CoreCLR_checked-(Ubuntu.1804.Arm32.Open)Ubuntu.1804.Armarch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7-bfcd90a-20200121150440 console.log core dump runclient.py
1227837 runtime PR 55372 net6.0-Linux-Release-arm-CoreCLR_checked-(Alpine.313.Arm32.Open)Ubuntu.1804.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:alpine-3.13-helix-arm32v7-20210414141857-1ea6b0a console.log core dump runclient.py
1227837 runtime PR 55372 net6.0-Linux-Debug-x64-CoreCLR_checked-Ubuntu.1804.Amd64.Open console.log core dump runclient.py
1227837 runtime PR 55372 net6.0-Linux-Debug-x64-CoreCLR_checked-(Alpine.312.Amd64.Open)ubuntu.1604.amd64.open@mcr.microsoft.com/dotnet-buildtools/prereqs:alpine-3.12-helix-20200602002622-e06dc59 console.log core dump runclient.py
1227631 runtime PR 55367 net6.0-OSX-Debug-x64-Mono_release-OSX.1015.Amd64.Open console.log
1227631 runtime PR 55367 net6.0-OSX-Debug-x64-CoreCLR_release-OSX.1015.Amd64.Open console.log
1227625 runtime PR 55346 net6.0-OSX-Debug-x64-Mono_release-OSX.1015.Amd64.Open console.log
1227625 runtime PR 55346 net6.0-OSX-Debug-x64-CoreCLR_release-OSX.1015.Amd64.Open console.log
1227555 runtime PR 55279 net6.0-OSX-Debug-x64-CoreCLR_release-OSX.1015.Amd64.Open console.log
1227555 runtime PR 55279 net6.0-OSX-Debug-x64-Mono_release-OSX.1015.Amd64.Open console.log
1227539 runtime PR 55362 net6.0-OSX-Debug-x64-Mono_release-OSX.1015.Amd64.Open console.log
1227539 runtime PR 55362 net6.0-OSX-Debug-x64-CoreCLR_release-OSX.1015.Amd64.Open console.log
1227470 runtime PR 55295 net6.0-OSX-Debug-x64-Mono_release-OSX.1015.Amd64.Open console.log
1227470 runtime PR 55295 net6.0-OSX-Debug-x64-CoreCLR_release-OSX.1015.Amd64.Open console.log
1227447 runtime PR 55150 net6.0-OSX-Debug-x64-Mono_release-OSX.1015.Amd64.Open console.log
1227447 runtime PR 55150 net6.0-OSX-Debug-x64-CoreCLR_release-OSX.1015.Amd64.Open console.log

Build Result Summary

Day Hit Count Week Hit Count Month Hit Count
1 51 51
@dotnet-issue-labeler dotnet-issue-labeler bot added area-System.Net untriaged New issue has not been triaged by the area owner labels Jul 15, 2021
@ghost
Copy link

ghost commented Jul 15, 2021

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

Issue Details

Runfo Creating Tracking Issue (data being generated)

Author: runfoapp[bot]
Assignees: -
Labels:

area-System.Net, untriaged
Milestone: -

@ManickaP
Copy link
Member

There's several problems combined here.

1. Infra issue - some of the console logs without a dump

Tests:

=== TEST EXECUTION SUMMARY ===
   System.Net.Requests.Tests  Total: 1109, Errors: 0, Failed: 0, Skipped: 6, Time: 48.794s
/root/helix/work/workitem
----- end Thu Jul 15 00:15:40 UTC 2021 ----- exit code 0 ----------------------------------------------------------
exit code 0 means Exited Successfully
+ export _commandExitCode=0

Infra after the tests passing:

2021-07-15 00:18:53,404: INFO: 548316835384: packing_test_reporter(34): report_results: Packed 317813 bytes
Traceback (most recent call last):
  File "/root/helix/work/correlation/reporter/run.py", line 73, in <module>
    main()
  File "/root/helix/work/correlation/reporter/run.py", line 69, in main
    if os.path.isfile(packed_file):
  File "/usr/lib/python3.8/genericpath.py", line 30, in isfile
    st = os.stat(path)
TypeError: stat: path should be string, bytes, os.PathLike or integer, not method
+ exit 1
+ export '_commandExitCode=1'
+ chmod -R 777 /home/helixbot/dotnetbuild/dumps
+ exit 1

[END EXECUTION]
Exit Code:1

cc: @premun

2. Mono crashes - with dump and runtime is mono

Console.log

=================================================================
	Native Crash Reporting
=================================================================
Got a SIGABRT while executing native code. This usually indicates
a fatal error in the mono runtime or one of the native libraries 
used by your application.
=================================================================
...

cc: @marek-safar

3. CoreCLR crashes - #54580 draft pr

Different errors, looks like work in progress:

----- end Wed 07/14/2021  1:03:58.23 ----- exit code -1073740791 ----------------------------------------------------------

----- end Wed Jul 14 01:49:23 UTC 2021 ----- exit code 139 ----------------------------------------------------------
exit code 139 means SIGSEGV Illegal memory access. Deref invalid pointer, overrunning buffer, stack overflow etc. Core dumped.

cc: @jkoritzinsky

4. Seg fault in FTP tests - majority of CoreCLR with a dump

Two errors, console.log:

./RunTests.sh: line 162:    24 Segmentation fault      (core dumped) "$RUNTIME_PATH/dotnet" exec --runtimeconfig System.Net.Requests.Tests.runtimeconfig.json --depsfile System.Net.Requests.Tests.deps.json xunit.console.dll System.Net.Requests.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing $RSP_FILE
/root/helix/work/workitem
----- end Mon Jul 12 19:32:39 UTC 2021 ----- exit code 139 ----------------------------------------------------------
exit code 139 means SIGSEGV Illegal memory access. Deref invalid pointer, overrunning buffer, stack overflow etc. Core dumped.

Assert failure(PID 59 [0x0000003b], Thread: 139 [0x008b]): !"Heap contamination detected! HeapFree was called on a heap other than the one that memory was allocated from.\n" "Possible cause: you used new (executable) to allocate the memory, but didn't use DeleteExecutable() to free it."
    File: /__w/1/s/src/coreclr/utilcode/clrhost_nodependencies.cpp Line: 296
    Image: /root/helix/work/correlation/dotnet

./RunTests.sh: line 162:    59 Aborted                 (core dumped) "$RUNTIME_PATH/dotnet" exec --runtimeconfig System.Net.Requests.Tests.runtimeconfig.json --depsfile System.Net.Requests.Tests.deps.json xunit.console.dll System.Net.Requests.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing $RSP_FILE

This is the first that looks like it might be System.Net problem. Or it could have been caused by #54954, since this PR has higher occurrence of this error.
cc: @karelz @janvorli

5. Only on PR #55467

Looks like work in progress, console.log:

Unhandled exception. System.IO.FileLoadException: Could not load file or assembly 'xunit.runner.utility.netcoreapp10, Version=2.4.1.0, Culture=neutral, PublicKeyToken=8d05b1bb7a6fdb6c'. Uncaught exception during type initialization. (0x80131534)
File name: 'xunit.runner.utility.netcoreapp10, Version=2.4.1.0, Culture=neutral, PublicKeyToken=8d05b1bb7a6fdb6c'
 ---> System.TypeInitializationException: The type initializer for 'Xunit.DependencyContextAssemblyCache' threw an exception.
 ---> System.NullReferenceException: Object reference not set to an instance of an object.
   at System.Text.RegularExpressions.RegexParser.AssignNameSlots() in /_/src/libraries/System.Text.RegularExpressions/src/System/Text/RegularExpressions/RegexParser.cs:line 1957
   at System.Text.RegularExpressions.RegexParser.CountCaptures() in /_/src/libraries/System.Text.RegularExpressions/src/System/Text/RegularExpressions/RegexParser.cs:line 1907
   at System.Text.RegularExpressions.RegexParser.Parse(String pattern, RegexOptions options, CultureInfo culture) in /_/src/libraries/System.Text.RegularExpressions/src/System/Text/RegularExpressions/RegexParser.cs:line 88
   at System.Text.RegularExpressions.Regex.Init(String pattern, RegexOptions options, TimeSpan matchTimeout, CultureInfo culture) in /_/src/libraries/System.Text.RegularExpressions/src/System/Text/RegularExpressions/Regex.cs:line 107
   at System.Text.RegularExpressions.Regex..ctor(String pattern, CultureInfo culture) in /_/src/libraries/System.Text.RegularExpressions/src/System/Text/RegularExpressions/Regex.cs:line 68
   at System.Text.RegularExpressions.Regex..ctor(String pattern) in /_/src/libraries/System.Text.RegularExpressions/src/System/Text/RegularExpressions/Regex.cs:line 46
   at Xunit.DependencyContextAssemblyCache..cctor() in /_/src/Microsoft.DotNet.XUnitConsoleRunner/src/common/AssemblyResolution/DependencyContextAssemblyCache.cs:line 20
   --- End of inner exception stack trace ---
   at Xunit.DependencyContextAssemblyCache.ResolveManagedAssembly(String assemblyName, Func`2 managedAssemblyLoader) in /_/src/Microsoft.DotNet.XUnitConsoleRunner/src/common/AssemblyResolution/DependencyContextAssemblyCache.cs:line 210
   at Xunit.DependencyContextAssemblyCache.LoadManagedDll(String assemblyName, Func`2 managedAssemblyLoader) in /_/src/Microsoft.DotNet.XUnitConsoleRunner/src/common/AssemblyResolution/DependencyContextAssemblyCache.cs:line 159
   at Xunit.AssemblyHelper.OnResolving(AssemblyLoadContext context, AssemblyName name) in /_/src/Microsoft.DotNet.XUnitConsoleRunner/src/common/AssemblyResolution/AssemblyHelper.cs:line 83
   at System.Runtime.Loader.AssemblyLoadContext.GetFirstResolvedAssemblyFromResolvingEvent(AssemblyName assemblyName) in /_/src/libraries/System.Private.CoreLib/src/System/Runtime/Loader/AssemblyLoadContext.cs:line 625
   at System.Runtime.Loader.AssemblyLoadContext.ResolveUsingEvent(AssemblyName assemblyName) in /_/src/libraries/System.Private.CoreLib/src/System/Runtime/Loader/AssemblyLoadContext.cs:line 693
   at System.Runtime.Loader.AssemblyLoadContext.ResolveUsingResolvingEvent(IntPtr gchManagedAssemblyLoadContext, AssemblyName assemblyName) in /_/src/coreclr/System.Private.CoreLib/src/System/Runtime/Loader/AssemblyLoadContext.CoreCLR.cs:line 134
   at Xunit.ConsoleClient.ConsoleRunner..ctor(Object consoleLock)
   at Xunit.ConsoleClient.Program.Main(String[] args) in /_/src/Microsoft.DotNet.XUnitConsoleRunner/src/Program.cs:line 16

cc: @eiriktsarpalis

6. Another CoreCLR crash, only #55372 PR

Happened only on this PR, so seems like fixed before it was merged, console.log:

Assert failure(PID 23 [0x00000017], Thread: 33 [0x0021]): At least one interface has to implement the method, otherwise there's a bug in JIT/verification.
    File: /__w/1/s/src/coreclr/vm/methodtable.cpp Line: 9578
    Image: /root/helix/work/correlation/dotnet

./RunTests.sh: line 162:    23 Aborted                 (core dumped) "$RUNTIME_PATH/dotnet" exec --runtimeconfig System.Net.Requests.Tests.runtimeconfig.json --depsfile System.Net.Requests.Tests.deps.json xunit.console.dll System.Net.Requests.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing $RSP_FILE

7. The last batch without a dump

CI timeout, console.log:

If you’re reading this, that means the Helix work item you’re trying to find the logs for has dead-lettered.

Looks like vast majority of problems is not networking related.

@janvorli
Copy link
Member

I have fixed the issue 4 from this list yesterday, the fix is already in.

@janvorli
Copy link
Member

I mean, the 2nd one in the issue 4.

@karelz
Copy link
Member

karelz commented Jul 15, 2021

@jaredpar this seems to be mix of many issues. We will take out the one crash that may be Networking related.
How do you want to track the rest?

Also, it seems that Runfo should not mix up the crashes as it did, but perhaps that is too much to ask from a simple tool?

@jaredpar
Copy link
Member

Also, it seems that Runfo should not mix up the crashes as it did, but perhaps that is too much to ask from a simple tool?

The tool right now only has indexes over certain strings: test names and issues in the timeline. Can design queries for pretty much anything along those lines.

The detail here to differentiate is in the console logs. That doesn't have any indexes yet. It's possible to do just havent' had time to do so.

@karelz
Copy link
Member

karelz commented Jul 20, 2021

Closing this mega-issue in favor of folks tracking down sub-problems as they are tagged above ...

@karelz karelz closed this as completed Jul 20, 2021
@karelz karelz added this to the 6.0.0 milestone Jul 20, 2021
@ghost ghost locked as resolved and limited conversation to collaborators Aug 19, 2021
@karelz karelz removed the untriaged New issue has not been triaged by the area owner label Oct 20, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
area-System.Net
Projects
None yet
Milestone
6.0.0
Development

No branches or pull requests
4 participants
@jaredpar @karelz @janvorli @ManickaP