Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

System.Net.Requests.Tests, System.Net.Security.Tests failing with Segmentation fault on Debian 10 #87268

Closed
ivanpovazan opened this issue Jun 8, 2023 · 5 comments · Fixed by #87288
Closed

System.Net.Requests.Tests, System.Net.Security.Tests failing with Segmentation fault on Debian 10 #87268

ivanpovazan opened this issue Jun 8, 2023 · 5 comments · Fixed by #87288
Labels
area-System.Net blocking-clean-ci Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms' Known Build Error Use this to report build issues in the .NET Helix tab
Milestone
8.0.0

Comments

@ivanpovazan
Copy link
Member

ivanpovazan commented Jun 8, 2023 •
edited by build-analysis bot
Loading

Build Information

Build: https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=298055

Build error legs:

  • linux-x64 Release NativeAOT_Checked_Libs_SizeOpt
  • linux-x64 Release NativeAOT_Checked_Libs_SpeedOpt

Tests failing:

  • System.Net.Requests.Tests
  • System.Net.Security.Tests

Pull request: #86652

Error Message

Fill the error message using step by step known issues guidance.

{
  "ErrorMessage": "",
  "ErrorPattern": "25 Segmentation fault      (core dumped) ./System.Net.Requests.Tests",
  "BuildRetry": false,
  "ExcludeConsoleLog": false
}

Report

Build Definition Step Name Console log Pull Request
299406 dotnet/runtime Build and Package Log #87148
299407 dotnet/runtime Build product Log #87148
Build Definition Test Pull Request
299800 dotnet/runtime System.Net.Security.Tests.WorkItemExecution
299568 dotnet/runtime JIT.opt.WorkItemExecution
299541 dotnet/runtime JIT.opt.WorkItemExecution #87261
299567 dotnet/runtime System.Net.Security.Tests.WorkItemExecution
299732 dotnet/runtime System.Net.Security.Tests.WorkItemExecution
299728 dotnet/runtime System.Net.Security.Tests.WorkItemExecution
299741 dotnet/runtime System.Net.Security.Tests.WorkItemExecution
299742 dotnet/runtime System.Net.Security.Tests.WorkItemExecution
299731 dotnet/runtime System.Net.Security.Tests.WorkItemExecution
299730 dotnet/runtime System.Net.Security.Tests.WorkItemExecution
299729 dotnet/runtime System.Net.Security.Tests.WorkItemExecution
299727 dotnet/runtime System.Net.Security.Tests.WorkItemExecution
299720 dotnet/runtime System.Net.Security.Tests.WorkItemExecution
299718 dotnet/runtime System.Net.Security.Tests.WorkItemExecution
299592 dotnet/runtime System.Net.Security.Tests.WorkItemExecution #86843
298109 dotnet/runtime System.Net.Security.Tests.WorkItemExecution #86578
299527 dotnet/runtime System.Net.Security.Tests.WorkItemExecution #86546
297387 dotnet/runtime System.Net.Security.Tests.WorkItemExecution #87189
299514 dotnet/runtime System.Net.Security.Tests.WorkItemExecution
299429 dotnet/runtime JIT.opt.WorkItemExecution
299427 dotnet/runtime JIT.opt.WorkItemExecution
299396 dotnet/runtime JIT.opt.WorkItemExecution
298623 dotnet/runtime System.Net.Security.Tests.WorkItemExecution #86759
299397 dotnet/runtime JIT.opt.WorkItemExecution
299365 dotnet/runtime System.Net.Security.Tests.WorkItemExecution #87204
299242 dotnet/runtime JIT.opt.WorkItemExecution
298991 dotnet/runtime System.Net.Security.Tests.WorkItemExecution #87209
299244 dotnet/runtime System.Net.Security.Tests.WorkItemExecution #87099
299225 dotnet/runtime System.Net.Security.Tests.WorkItemExecution #87198
299206 dotnet/runtime System.Net.Security.Tests.WorkItemExecution #87065
299181 dotnet/runtime System.Net.Security.Tests.WorkItemExecution #87237
298055 dotnet/runtime System.Net.Security.Tests.WorkItemExecution #86652
299037 dotnet/runtime System.Net.Security.Tests.WorkItemExecution
298966 dotnet/runtime System.Net.Security.Tests.WorkItemExecution #87233
298965 dotnet/runtime System.Net.Security.Tests.WorkItemExecution #87232
298677 dotnet/runtime JIT.opt.WorkItemExecution
298811 dotnet/runtime System.Net.Security.Tests.WorkItemExecution
298918 dotnet/runtime System.Runtime.Tests.WorkItemExecution #87230
298816 dotnet/runtime System.Net.Security.Tests.WorkItemExecution
298414 dotnet/runtime ComInterfaceGenerator.Unit.Tests.WorkItemExecution #87125
297910 dotnet/runtime JIT.opt.WorkItemExecution

Summary

24-Hour Hit Count 7-Day Hit Count 1-Month Count
35 43 43
@ivanpovazan ivanpovazan added blocking-clean-ci Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms' area-NativeAOT-coreclr Known Build Error Use this to report build issues in the .NET Helix tab labels Jun 8, 2023
@ghost ghost added the untriaged New issue has not been triaged by the area owner label Jun 8, 2023
@ghost
Copy link

ghost commented Jun 8, 2023

Tagging subscribers to this area: @agocke, @MichalStrehovsky, @jkotas
See info in area-owners.md if you want to be subscribed.

Issue Details

Build Information

Build: https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=298055

Build error legs:

  • linux-x64 Release NativeAOT_Checked_Libs_SizeOpt
  • linux-x64 Release NativeAOT_Checked_Libs_SpeedOpt

Tests failing:

  • System.Net.Quic.Functional.Tests
  • System.Net.Requests.Tests
  • System.Net.Security.Tests

Pull request: #86652

Error Message

Fill the error message using step by step known issues guidance.

{
  "ErrorMessage": "Segmentation fault",
  "ErrorPattern": "",
  "BuildRetry": false,
  "ExcludeConsoleLog": false
}
Author: ivanpovazan
Assignees: -
Labels:

blocking-clean-ci, area-NativeAOT-coreclr, Known Build Error
Milestone: -

@EgorBo
Copy link
Member

EgorBo commented Jun 8, 2023

I'll check whether it could be #87115

@ivanpovazan
Copy link
Member Author

Updated the search term to use ErrorPattern instead, as the initial "ErrorMessage": "Segmentation fault" was too broad

@jkotas
Copy link
Member

jkotas commented Jun 8, 2023 •
edited
Loading

https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=298055

System.Net.Quic.Functional.Tests is known issue #87275.

System.Net.Requests.Tests is crashing is crashing in libcrypto.so.1.1 initialization on Debian 10. I do not think it has anything to do with the size/speed opt configs, we just happened to hit this problem in these configs first.

Unfortunately, it is not possible to get accurate stacktrace to debug this further since Debian does not publish symbols for security updates (https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=894081).

We do not support Debian 10 for .NET 8 according to https://github.com/dotnet/core/blob/main/release-notes/8.0/supported-os.md#linux . We should "fix this" by deleting Debian 10 test runs for .NET 8.

#0  0x000000000008bb56 in ?? ()
#1  0x00007f8f8aea1cb0 in ?? () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1
#2  0x00007f8f8ae30017 in ?? () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1
#3  0x00007f8f88d18f48 in ?? () from /usr/lib/x86_64-linux-gnu/libssl.so.1.1
#4  0x00007fd02726e34a in call_init (l=0x563d7fe57920, argc=argc@entry=9, argv=argv@entry=0x7fffd8bbf948, env=env@entry=0x7fffd8bbf998) at dl-init.c:58
#5  0x00007fd02726e486 in call_init (env=0x7fffd8bbf998, argv=0x7fffd8bbf948, argc=9, l=<optimized out>) at dl-init.c:30
#6  _dl_init (main_map=main_map@entry=0x7f8f7800a270, argc=9, argv=0x7fffd8bbf948, env=0x7fffd8bbf998) at dl-init.c:119
#7  0x00007fd0272722e3 in dl_open_worker (a=a@entry=0x7f8f895a3d80) at dl-open.c:517
#8  0x00007fd026ffa8cf in __GI__dl_catch_exception (exception=exception@entry=0x7f8f895a3d60, operate=operate@entry=0x7fd027271f70 <dl_open_worker>, args=args@entry=0x7f8f895a3d80) at dl-error-skeleton.c:196
#9  0x00007fd027271bca in _dl_open (file=0x563d7d7efebb "libssl.so.1.1", mode=-2147483647, caller_dlopen=0x563d7dc62c5c, nsid=<optimized out>, argc=9, argv=0x7fffd8bbf948, env=0x7fffd8bbf998) at dl-open.c:599
#10 0x00007fd027253256 in dlopen_doit (a=a@entry=0x7f8f895a3fa0) at dlopen.c:66
#11 0x00007fd026ffa8cf in __GI__dl_catch_exception (exception=exception@entry=0x7f8f895a3f40, operate=operate@entry=0x7fd027253200 <dlopen_doit>, args=args@entry=0x7f8f895a3fa0) at dl-error-skeleton.c:196
#12 0x00007fd026ffa95f in __GI__dl_catch_error (objname=objname@entry=0x7f8f78009170, errstring=errstring@entry=0x7f8f78009178, mallocedp=mallocedp@entry=0x7f8f78009168,
    operate=operate@entry=0x7fd027253200 <dlopen_doit>, args=args@entry=0x7f8f895a3fa0) at dl-error-skeleton.c:215
#13 0x00007fd027253995 in _dlerror_run (operate=operate@entry=0x7fd027253200 <dlopen_doit>, args=args@entry=0x7f8f895a3fa0) at dlerror.c:170
#14 0x00007fd0272532e6 in __dlopen (file=<optimized out>, mode=<optimized out>) at dlopen.c:87
#15 0x0000563d7dc62c5c in ?? ()
#16 0x00007fd027095997 in __pthread_once_slow (once_control=0x563d7e618b68, init_routine=0x563d7dc62b80) at pthread_once.c:116
#17 0x0000563d7dc62d99 in InitializeOpenSSLShim ()
#18 0x0000563d7dc5bd8f in ?? ()
#19 0x00007fd027095997 in __pthread_once_slow (once_control=0x563d7e618b2c, init_routine=0x563d7dc5bd80) at pthread_once.c:116
#20 0x0000563d7dc5bd77 in CryptoNative_EnsureOpenSslInitialized ()
#21 0x0000563d7df5a4df in TestUtilities_Interop_CryptoInitializer__EnsureOpenSslInitialized ()
#22 0x0000563d7df5a469 in TestUtilities_Interop_CryptoInitializer___cctor ()
#23 0x0000563d7dee317e in S_P_CoreLib_System_Runtime_CompilerServices_ClassConstructorRunner__EnsureClassConstructorRun ()
#24 0x0000563d7dee309d in S_P_CoreLib_System_Runtime_CompilerServices_ClassConstructorRunner__CheckStaticClassConstructionReturnNonGCStaticBase ()
#25 0x0000563d7df5a3a9 in TestUtilities_Interop_OpenSsl___cctor ()
#26 0x0000563d7dee317e in S_P_CoreLib_System_Runtime_CompilerServices_ClassConstructorRunner__EnsureClassConstructorRun ()
#27 0x0000563d7dee309d in S_P_CoreLib_System_Runtime_CompilerServices_ClassConstructorRunner__CheckStaticClassConstructionReturnNonGCStaticBase ()
#28 0x0000563d7df5a3fe in TestUtilities_Interop_OpenSsl__OpenSslVersionNumber ()
#29 0x0000563d7df5487c in TestUtilities_System_PlatformDetection__GetOpenSslVersion ()
#30 0x0000563d7df54499 in TestUtilities_System_PlatformDetection__get_OpenSslVersion ()

@jkotas jkotas changed the title [NativeAOT] Tests failing with Segmentation fault with Size/Speed optimization config System.Net.Requests.Tests, System.Net.Security.Tests failing with Segmentation fault on Debian 10 Jun 8, 2023
@ghost
Copy link

ghost commented Jun 8, 2023

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

Issue Details

Build Information

Build: https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=298055

Build error legs:

  • linux-x64 Release NativeAOT_Checked_Libs_SizeOpt
  • linux-x64 Release NativeAOT_Checked_Libs_SpeedOpt

Tests failing:

  • System.Net.Quic.Functional.Tests
  • System.Net.Requests.Tests
  • System.Net.Security.Tests

Pull request: #86652

Error Message

Fill the error message using step by step known issues guidance.

{
  "ErrorMessage": "",
  "ErrorPattern": "RunTests\\.sh.+Segmentation fault",
  "BuildRetry": false,
  "ExcludeConsoleLog": false
}

Report

Build Definition Step Name Console log Pull Request
299406 dotnet/runtime Build and Package Log #87148
299407 dotnet/runtime Build product Log #87148
Build Definition Test Pull Request
299800 dotnet/runtime System.Net.Security.Tests.WorkItemExecution
299568 dotnet/runtime JIT.opt.WorkItemExecution
299541 dotnet/runtime JIT.opt.WorkItemExecution #87261
299567 dotnet/runtime System.Net.Security.Tests.WorkItemExecution
299732 dotnet/runtime System.Net.Security.Tests.WorkItemExecution
299728 dotnet/runtime System.Net.Security.Tests.WorkItemExecution
299741 dotnet/runtime System.Net.Security.Tests.WorkItemExecution
299742 dotnet/runtime System.Net.Security.Tests.WorkItemExecution
299731 dotnet/runtime System.Net.Security.Tests.WorkItemExecution
299730 dotnet/runtime System.Net.Security.Tests.WorkItemExecution
299729 dotnet/runtime System.Net.Security.Tests.WorkItemExecution
299727 dotnet/runtime System.Net.Security.Tests.WorkItemExecution
299720 dotnet/runtime System.Net.Security.Tests.WorkItemExecution
299718 dotnet/runtime System.Net.Security.Tests.WorkItemExecution
299592 dotnet/runtime System.Net.Security.Tests.WorkItemExecution #86843
298109 dotnet/runtime System.Net.Security.Tests.WorkItemExecution #86578
299527 dotnet/runtime System.Net.Security.Tests.WorkItemExecution #86546
297387 dotnet/runtime System.Net.Security.Tests.WorkItemExecution #87189
299514 dotnet/runtime System.Net.Security.Tests.WorkItemExecution
299429 dotnet/runtime JIT.opt.WorkItemExecution
299427 dotnet/runtime JIT.opt.WorkItemExecution
299396 dotnet/runtime JIT.opt.WorkItemExecution
298623 dotnet/runtime System.Net.Security.Tests.WorkItemExecution #86759
299397 dotnet/runtime JIT.opt.WorkItemExecution
299365 dotnet/runtime System.Net.Security.Tests.WorkItemExecution #87204
299242 dotnet/runtime JIT.opt.WorkItemExecution
298991 dotnet/runtime System.Net.Security.Tests.WorkItemExecution #87209
299244 dotnet/runtime System.Net.Security.Tests.WorkItemExecution #87099
299225 dotnet/runtime System.Net.Security.Tests.WorkItemExecution #87198
299206 dotnet/runtime System.Net.Security.Tests.WorkItemExecution #87065
299181 dotnet/runtime System.Net.Security.Tests.WorkItemExecution #87237
298055 dotnet/runtime System.Net.Security.Tests.WorkItemExecution #86652
299037 dotnet/runtime System.Net.Security.Tests.WorkItemExecution
298966 dotnet/runtime System.Net.Security.Tests.WorkItemExecution #87233
298965 dotnet/runtime System.Net.Security.Tests.WorkItemExecution #87232
298677 dotnet/runtime JIT.opt.WorkItemExecution
298811 dotnet/runtime System.Net.Security.Tests.WorkItemExecution
298918 dotnet/runtime System.Runtime.Tests.WorkItemExecution #87230
298816 dotnet/runtime System.Net.Security.Tests.WorkItemExecution
298414 dotnet/runtime ComInterfaceGenerator.Unit.Tests.WorkItemExecution #87125
297910 dotnet/runtime JIT.opt.WorkItemExecution

Summary

24-Hour Hit Count 7-Day Hit Count 1-Month Count
41 43 43
Author: ivanpovazan
Assignees: -
Labels:

area-System.Net, blocking-clean-ci, untriaged, area-NativeAOT-coreclr, Known Build Error
Milestone: -

jkotas added a commit to jkotas/runtime that referenced this issue Jun 8, 2023
@jkotas
Delete Debian 10 test runs
88c9626 
Debian 10 is not supported for .NET 8.

Fixes dotnet#87268
@jkotas jkotas mentioned this issue Jun 8, 2023
Merged
@ghost ghost added the in-pr There is an active PR which will close this issue when it is merged label Jun 8, 2023
@Maoni0 Maoni0 mentioned this issue Jun 8, 2023
Closed
jkotas added a commit that referenced this issue Jun 9, 2023
@jkotas
Delete Debian 10 test runs (#87288)
9bcf523 
Debian 10 is not supported for .NET 8.

Fixes #87268
@ghost ghost removed in-pr There is an active PR which will close this issue when it is merged untriaged New issue has not been triaged by the area owner labels Jun 9, 2023
@Maoni0 Maoni0 mentioned this issue Jun 9, 2023
Closed
@karelz karelz added this to the 8.0.0 milestone Jul 3, 2023
@ghost ghost locked as resolved and limited conversation to collaborators Aug 2, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
area-System.Net blocking-clean-ci Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms' Known Build Error Use this to report build issues in the .NET Helix tab
Projects
None yet
Milestone
8.0.0
Development

Successfully merging a pull request may close this issue.

Delete Debian 10 test runs jkotas/runtime
4 participants
@EgorBo @karelz @jkotas @ivanpovazan