Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

MsQuic tests hang / long running #56487

Closed
karelz opened this issue Jul 28, 2021 · 5 comments
Closed

MsQuic tests hang / long running #56487

karelz opened this issue Jul 28, 2021 · 5 comments
Labels
area-System.Net.Quic test-run-core Test failures in .NET Core test runs
Projects
HTTP/3
Milestone
6.0.0

Comments

@karelz
Copy link
Member

karelz commented Jul 28, 2021 •
edited
Loading

Affected tests:

  • System.Net.Quic.Tests.MsQuicQuicStreamConformanceTests:
    • ReadWrite_Success
    • ReadWrite_MessagesSmallerThanReadBuffer_Success
  • System.Net.Quic.Tests.QuicStreamTests_MsQuicProvider:
    • ReadWrite_Random_Success
    • LargeDataSentAndReceived
    • MultipleReadsAndWrites

Failures 3/29-7/28 (incl. PRs):

  • Always at least 2 tests -- mostly 2 tests hang at the same time
  • Most hangs are on Mono
  • Looks like a regression on 7/13
Day Run Runtime Test 1 (MsQuicQuicStreamConformanceTests) Test 2 (QuicStreamTests_MsQuicProvider) Additional tests
7/13 PR #55550 Mono ReadWrite_MessagesSmallerThanReadBuffer_Success ReadWrite_Random_Success 3rd: sQuicTests.ByteMixingOrNativeAVE_MinimalFailingTest
7/13 PR #55332 Mono ReadWrite_MessagesSmallerThanReadBuffer_Success ReadWrite_Random_Success 3rd: MsQuicTests.ByteMixingOrNativeAVE_MinimalFailingTest
7/14 PR #55599 Mono ReadWrite_MessagesSmallerThanReadBuffer_Success ReadWrite_Random_Success 3rd: MsQuicTests.ByteMixingOrNativeAVE_MinimalFailingTest
7/14 PR #54293 Mono ReadWrite_Success ReadWrite_Random_Success --
7/14 PR #55533 Mono ReadWrite_MessagesSmallerThanReadBuffer_Success ReadWrite_Random_Success --
7/14 PR #55533 CoreCLR ReadWrite_Success LargeDataSentAndReceived --
7/14 PR #55594 Mono ReadWrite_Success ReadWrite_Random_Success --
7/14 Official run Mono ReadWrite_Success ReadWrite_Random_Success --
7/14 PR #55613 Mono ReadWrite_Success ReadWrite_Random_Success --
7/14 PR #55636 Mono ReadWrite_Success LargeDataSentAndReceived --
7/14 PR #55648 Mono ReadWrite_Success ReadWrite_Random_Success --
7/14 PR #54453 Mono ReadWrite_Success ReadWrite_Random_Success --
7/14 PR #55667 Mono ReadWrite_Success ReadWrite_Random_Success --
7/14 PR #55596 Mono ReadWrite_Success ReadWrite_Random_Success --
7/14 PR #54640 Mono ReadWrite_Success ReadWrite_Random_Success --
7/15 PR #52244 Mono ReadWrite_Success ReadWrite_Random_Success --
7/15 Official run Mono ReadWrite_Success MultipleReadsAndWrites --
7/15 PR #55719 Mono ReadWrite_Success ReadWrite_Random_Success --
7/15 PR #53505 Mono ReadWrite_Success ReadWrite_Random_Success --
7/15 PR #55658 Mono ReadWrite_MessagesSmallerThanReadBuffer_Success ReadWrite_Random_Success --
7/15 PR #55710 Mono ReadWrite_Success ReadWrite_Random_Success --
7/15 PR #52699 Mono ReadWrite_Success MultipleReadsAndWrites --
7/16 PR #55776 Mono ReadWrite_MessagesSmallerThanReadBuffer_Success ReadWrite_Random_Success --
7/16 PR #55772 Mono ReadWrite_Success MultipleReadsAndWrites --
7/16 PR #55724 Mono ReadWrite_Success ReadWrite_Random_Success --
7/16 PR #54640 Mono ReadWrite_MessagesSmallerThanReadBuffer_Success ReadWrite_Random_Success --
7/16 PR #55801 Mono ReadWrite_Success ReadWrite_Random_Success --
7/16 PR #55787 Mono ReadWrite_Success ReadWrite_Random_Success --
7/16 PR #55769 Mono ReadWrite_Success ReadWrite_Random_Success --
7/16 Official run CoreCLR ReadWrite_Success LargeDataSentAndReceived --
7/16 PR #55797 Mono ReadWrite_MessagesSmallerThanReadBuffer_Success ReadWrite_Random_Success --
7/17 PR #55875 Mono ReadWrite_Success ReadWrite_Random_Success --
7/17 PR #55875 Mono ReadWrite_Success ReadWrite_Random_Success --
7/18 Official run Mono ReadWrite_Success LargeDataSentAndReceived --
7/18 PR #55877 Mono ReadWrite_Success ReadWrite_Random_Success --
7/19 PR #54640 Mono ReadWrite_Success ReadWrite_Random_Success --
7/19 PR #55912 Mono ReadWrite_MessagesSmallerThanReadBuffer_Success ReadWrite_Random_Success --
7/19 Official run Mono ReadWrite_Success ReadWrite_Random_Success --
7/19 PR #55883 Mono ReadWrite_MessagesSmallerThanReadBuffer_Success ReadWrite_Random_Success --
7/19 PR #54640 Mono ReadWrite_Success ReadWrite_Random_Success --
7/19 PR #55925 Mono ReadWrite_Success LargeDataSentAndReceived --
7/19 PR #55925 CoreCLR ReadWrite_MessagesSmallerThanReadBuffer_Success ReadWrite_Random_Success --
7/19 PR #55942 Mono ReadWrite_Success ReadWrite_Random_Success --
7/19 PR #55915 Mono ReadWrite_Success MultipleReadsAndWrites --
7/19 PR #55862 Mono ReadWrite_Success ReadWrite_Random_Success --
7/19 PR #55283 Mono ReadWrite_Success MultipleReadsAndWrites 3rd: QuicStreamTests_MockProvider.ReadWrite_Random_Success
7/19 PR #55871 Mono ReadWrite_Success ReadWrite_Random_Success --
7/19 Official run Mono ReadWrite_Success ReadWrite_Random_Success --
7/20 PR #55564 Mono ReadWrite_Success ReadWrite_Random_Success --
7/20 PR #56003 Mono ReadWrite_Success MultipleReadsAndWrites 3rd: QuicStreamTests_MockProvider.ReadWrite_Random_Success
4th: MockQuicStreamConformanceTests.CopyToAsync_AllDataCopied
7/20 PR #55871 Mono ReadWrite_MessagesSmallerThanReadBuffer_Success ReadWrite_Random_Success --
7/20 PR #56030 Mono ReadWrite_Success ReadWrite_Random_Success --
7/20 PR #56039 Mono ReadWrite_MessagesSmallerThanReadBuffer_Success ReadWrite_Random_Success --
7/22 PR #56128 Mono ReadWrite_Success LargeDataSentAndReceived --
7/22 PR #56130 Mono ReadWrite_Success ReadWrite_Random_Success --
7/22 PR #56128 Mono ReadWrite_Success LargeDataSentAndReceived --
7/22 PR #56130 Mono ReadWrite_Success ReadWrite_Random_Success --
7/23 PR #56235 Mono ReadWrite_MessagesSmallerThanReadBuffer_Success ReadWrite_Random_Success --
7/23 PR #56235 Mono ReadWrite_Success ReadWrite_Random_Success --
7/26 PR #56318 Mono ReadWrite_Success LargeDataSentAndReceived --
7/27 PR #56330 CoreCLR ReadWrite_Success LargeDataSentAndReceived --
7/28 PR #56433 Mono ReadWrite_Success ReadWrite_Random_Success --

Example from 7/28:

===========================================================================================================
/root/helix/work/workitem /root/helix/work/workitem
  Discovering: System.Net.Quic.Functional.Tests (method display = ClassAndMethod, method display options = None)
  Discovered:  System.Net.Quic.Functional.Tests (found 104 of 121 test cases)
  Starting:    System.Net.Quic.Functional.Tests (parallel test collections = on, max threads = 2)
   System.Net.Quic.Functional.Tests: [Long Running Test] 'System.Net.Quic.Tests.QuicStreamTests_MsQuicProvider.ReadWrite_Random_Success', Elapsed: 00:02:10
[Long Running Test] 'System.Net.Quic.Tests.MsQuicQuicStreamConformanceTests.ReadWrite_Success', Elapsed: 00:02:16
   System.Net.Quic.Functional.Tests: [Long Running Test] 'System.Net.Quic.Tests.QuicStreamTests_MsQuicProvider.ReadWrite_Random_Success', Elapsed: 00:04:10
[Long Running Test] 'System.Net.Quic.Tests.MsQuicQuicStreamConformanceTests.ReadWrite_Success', Elapsed: 00:04:16
   System.Net.Quic.Functional.Tests: [Long Running Test] 'System.Net.Quic.Tests.QuicStreamTests_MsQuicProvider.ReadWrite_Random_Success', Elapsed: 00:06:10
[Long Running Test] 'System.Net.Quic.Tests.MsQuicQuicStreamConformanceTests.ReadWrite_Success', Elapsed: 00:06:16
   System.Net.Quic.Functional.Tests: [Long Running Test] 'System.Net.Quic.Tests.QuicStreamTests_MsQuicProvider.ReadWrite_Random_Success', Elapsed: 00:08:10
[Long Running Test] 'System.Net.Quic.Tests.MsQuicQuicStreamConformanceTests.ReadWrite_Success', Elapsed: 00:08:16
   System.Net.Quic.Functional.Tests: [Long Running Test] 'System.Net.Quic.Tests.QuicStreamTests_MsQuicProvider.ReadWrite_Random_Success', Elapsed: 00:10:10
[Long Running Test] 'System.Net.Quic.Tests.MsQuicQuicStreamConformanceTests.ReadWrite_Success', Elapsed: 00:10:16
   System.Net.Quic.Functional.Tests: [Long Running Test] 'System.Net.Quic.Tests.QuicStreamTests_MsQuicProvider.ReadWrite_Random_Success', Elapsed: 00:12:10
[Long Running Test] 'System.Net.Quic.Tests.MsQuicQuicStreamConformanceTests.ReadWrite_Success', Elapsed: 00:12:16
   System.Net.Quic.Functional.Tests: [Long Running Test] 'System.Net.Quic.Tests.QuicStreamTests_MsQuicProvider.ReadWrite_Random_Success', Elapsed: 00:14:10
[Long Running Test] 'System.Net.Quic.Tests.MsQuicQuicStreamConformanceTests.ReadWrite_Success', Elapsed: 00:14:16

...
[EXECUTION TIMED OUT]
Exit Code:-3Executor timed out after 900 seconds and was killed
['System.Net.Quic.Functional.Tests' END OF WORK ITEM LOG: Command timed out, and was killed]
@karelz karelz added this to the 6.0.0 milestone Jul 28, 2021
@karelz karelz added this to To Do (Low Priority) in HTTP/3 via automation Jul 28, 2021
@dotnet-issue-labeler dotnet-issue-labeler bot added the untriaged New issue has not been triaged by the area owner label Jul 28, 2021
@ghost
Copy link

ghost commented Jul 28, 2021

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

Issue Details

Affected tests:

  • System.Net.Quic.Tests.MsQuicQuicStreamConformanceTests:
    • ReadWrite_Success
    • ReadWrite_MessagesSmallerThanReadBuffer_Success
  • System.Net.Quic.Tests.QuicStreamTests_MsQuicProvider:
    • ReadWrite_Random_Success
    • LargeDataSentAndReceived
    • MultipleReadsAndWrites

Failures 3/29-7/28 (incl. PRs):

  • Always at least 2 tests -- mostly 2 tests hang at the same time
  • Most hangs are on Mono
  • Looks like a regression on 7/13
Day Run Runtime Test 1 (MsQuicQuicStreamConformanceTests) Test 2 (QuicStreamTests_MsQuicProvider) Additional tests
7/13 PR #55550 Mono ReadWrite_MessagesSmallerThanReadBuffer_Success ReadWrite_Random_Success MsQuicTests.ByteMixingOrNativeAVE_MinimalFailingTest
7/13 PR #55332 Mono ReadWrite_MessagesSmallerThanReadBuffer_Success ReadWrite_Random_Success MsQuicTests.ByteMixingOrNativeAVE_MinimalFailingTest
7/14 PR #55599 Mono ReadWrite_MessagesSmallerThanReadBuffer_Success ReadWrite_Random_Success MsQuicTests.ByteMixingOrNativeAVE_MinimalFailingTest
7/14 PR #54293 Mono ReadWrite_Success ReadWrite_Random_Success --
7/14 PR #55533 Mono ReadWrite_MessagesSmallerThanReadBuffer_Success ReadWrite_Random_Success --
7/14 PR #55533 CoreCLR ReadWrite_Success LargeDataSentAndReceived --
7/14 PR #55594 Mono ReadWrite_Success ReadWrite_Random_Success --
7/14 Official run Mono ReadWrite_Success ReadWrite_Random_Success --
7/14 PR #55613 Mono ReadWrite_Success ReadWrite_Random_Success --
7/14 PR #55636 Mono ReadWrite_Success LargeDataSentAndReceived --
7/14 PR #55648 Mono ReadWrite_Success ReadWrite_Random_Success --
7/14 PR #54453 Mono ReadWrite_Success ReadWrite_Random_Success --
7/14 PR #55667 Mono ReadWrite_Success ReadWrite_Random_Success --
7/14 PR #55596 Mono ReadWrite_Success ReadWrite_Random_Success --
7/14 PR #54640 Mono ReadWrite_Success ReadWrite_Random_Success --
7/15 PR #52244 Mono ReadWrite_Success ReadWrite_Random_Success --
7/15 Official run Mono ReadWrite_Success MultipleReadsAndWrites --
7/15 PR #55719 Mono ReadWrite_Success ReadWrite_Random_Success --
7/15 PR #53505 Mono ReadWrite_Success ReadWrite_Random_Success --
7/15 PR #55658 Mono ReadWrite_MessagesSmallerThanReadBuffer_Success ReadWrite_Random_Success --
7/15 PR #55710 Mono ReadWrite_Success ReadWrite_Random_Success --
7/15 PR #52699 Mono ReadWrite_Success MultipleReadsAndWrites --
7/16 PR #55776 Mono ReadWrite_MessagesSmallerThanReadBuffer_Success ReadWrite_Random_Success --
7/16 PR #55772 Mono ReadWrite_Success MultipleReadsAndWrites --
7/16 PR #55724 Mono ReadWrite_Success ReadWrite_Random_Success --
7/16 PR #54640 Mono ReadWrite_MessagesSmallerThanReadBuffer_Success ReadWrite_Random_Success --
7/16 PR #55801 Mono ReadWrite_Success ReadWrite_Random_Success --
7/16 PR #55787 Mono ReadWrite_Success ReadWrite_Random_Success --
7/16 PR #55769 Mono ReadWrite_Success ReadWrite_Random_Success --
7/16 Official run CoreCLR ReadWrite_Success LargeDataSentAndReceived --
7/16 PR #55797 Mono ReadWrite_MessagesSmallerThanReadBuffer_Success ReadWrite_Random_Success --
7/17 PR #55875 Mono ReadWrite_Success ReadWrite_Random_Success --
7/17 PR #55875 Mono ReadWrite_Success ReadWrite_Random_Success --
7/18 Official run Mono ReadWrite_Success LargeDataSentAndReceived --
7/18 PR #55877 Mono ReadWrite_Success ReadWrite_Random_Success --
7/19 PR #54640 Mono ReadWrite_Success ReadWrite_Random_Success --
7/19 PR #55912 Mono ReadWrite_MessagesSmallerThanReadBuffer_Success ReadWrite_Random_Success --
7/19 Official run Mono ReadWrite_Success ReadWrite_Random_Success --
7/19 PR #55883 Mono ReadWrite_MessagesSmallerThanReadBuffer_Success ReadWrite_Random_Success --
7/19 PR #54640 Mono ReadWrite_Success ReadWrite_Random_Success --
7/19 PR #55925 Mono ReadWrite_Success LargeDataSentAndReceived --
7/19 PR #55925 CoreCLR ReadWrite_MessagesSmallerThanReadBuffer_Success ReadWrite_Random_Success --
7/19 PR #55942 Mono ReadWrite_Success ReadWrite_Random_Success --
7/19 PR #55915 Mono ReadWrite_Success MultipleReadsAndWrites --
7/19 PR #55862 Mono ReadWrite_Success ReadWrite_Random_Success --
7/19 PR #55283 Mono ReadWrite_Success MultipleReadsAndWrites QuicStreamTests_MockProvider.ReadWrite_Random_Success
7/19 PR #55871 Mono ReadWrite_Success ReadWrite_Random_Success --
7/19 Official run Mono ReadWrite_Success ReadWrite_Random_Success --
7/20 PR #55564 Mono ReadWrite_Success ReadWrite_Random_Success --
7/20 PR #56003 Mono ReadWrite_Success MultipleReadsAndWrites QuicStreamTests_MockProvider.ReadWrite_Random_Success & MockQuicStreamConformanceTests.CopyToAsync_AllDataCopied
7/20 PR #55871 Mono ReadWrite_MessagesSmallerThanReadBuffer_Success ReadWrite_Random_Success --
7/20 PR #56030 Mono ReadWrite_Success ReadWrite_Random_Success --
7/20 PR #56039 Mono ReadWrite_MessagesSmallerThanReadBuffer_Success ReadWrite_Random_Success --
7/22 PR #56128 Mono ReadWrite_Success LargeDataSentAndReceived --
7/22 PR #56130 Mono ReadWrite_Success ReadWrite_Random_Success --
7/22 PR #56128 Mono ReadWrite_Success LargeDataSentAndReceived --
7/22 PR #56130 Mono ReadWrite_Success ReadWrite_Random_Success --
7/23 PR #56235 Mono ReadWrite_MessagesSmallerThanReadBuffer_Success ReadWrite_Random_Success --
7/23 PR #56235 Mono ReadWrite_Success ReadWrite_Random_Success --
7/26 PR #56318 Mono ReadWrite_Success LargeDataSentAndReceived --
7/27 PR #56330 CoreCLR ReadWrite_Success LargeDataSentAndReceived --
7/28 PR #56433 Mono ReadWrite_Success ReadWrite_Random_Success --

Example from 7/28:

===========================================================================================================
/root/helix/work/workitem /root/helix/work/workitem
  Discovering: System.Net.Quic.Functional.Tests (method display = ClassAndMethod, method display options = None)
  Discovered:  System.Net.Quic.Functional.Tests (found 104 of 121 test cases)
  Starting:    System.Net.Quic.Functional.Tests (parallel test collections = on, max threads = 2)
   System.Net.Quic.Functional.Tests: [Long Running Test] 'System.Net.Quic.Tests.QuicStreamTests_MsQuicProvider.ReadWrite_Random_Success', Elapsed: 00:02:10
[Long Running Test] 'System.Net.Quic.Tests.MsQuicQuicStreamConformanceTests.ReadWrite_Success', Elapsed: 00:02:16
   System.Net.Quic.Functional.Tests: [Long Running Test] 'System.Net.Quic.Tests.QuicStreamTests_MsQuicProvider.ReadWrite_Random_Success', Elapsed: 00:04:10
[Long Running Test] 'System.Net.Quic.Tests.MsQuicQuicStreamConformanceTests.ReadWrite_Success', Elapsed: 00:04:16
   System.Net.Quic.Functional.Tests: [Long Running Test] 'System.Net.Quic.Tests.QuicStreamTests_MsQuicProvider.ReadWrite_Random_Success', Elapsed: 00:06:10
[Long Running Test] 'System.Net.Quic.Tests.MsQuicQuicStreamConformanceTests.ReadWrite_Success', Elapsed: 00:06:16
   System.Net.Quic.Functional.Tests: [Long Running Test] 'System.Net.Quic.Tests.QuicStreamTests_MsQuicProvider.ReadWrite_Random_Success', Elapsed: 00:08:10
[Long Running Test] 'System.Net.Quic.Tests.MsQuicQuicStreamConformanceTests.ReadWrite_Success', Elapsed: 00:08:16
   System.Net.Quic.Functional.Tests: [Long Running Test] 'System.Net.Quic.Tests.QuicStreamTests_MsQuicProvider.ReadWrite_Random_Success', Elapsed: 00:10:10
[Long Running Test] 'System.Net.Quic.Tests.MsQuicQuicStreamConformanceTests.ReadWrite_Success', Elapsed: 00:10:16
   System.Net.Quic.Functional.Tests: [Long Running Test] 'System.Net.Quic.Tests.QuicStreamTests_MsQuicProvider.ReadWrite_Random_Success', Elapsed: 00:12:10
[Long Running Test] 'System.Net.Quic.Tests.MsQuicQuicStreamConformanceTests.ReadWrite_Success', Elapsed: 00:12:16
   System.Net.Quic.Functional.Tests: [Long Running Test] 'System.Net.Quic.Tests.QuicStreamTests_MsQuicProvider.ReadWrite_Random_Success', Elapsed: 00:14:10
[Long Running Test] 'System.Net.Quic.Tests.MsQuicQuicStreamConformanceTests.ReadWrite_Success', Elapsed: 00:14:16

...
[EXECUTION TIMED OUT]
Exit Code:-3Executor timed out after 900 seconds and was killed
['System.Net.Quic.Functional.Tests' END OF WORK ITEM LOG: Command timed out, and was killed]
Author: karelz
Assignees: -
Labels:

area-System.Net.Quic
Milestone: 6.0.0

@karelz karelz removed the untriaged New issue has not been triaged by the area owner label Jul 28, 2021
@wfurt
Copy link
Member

wfurt commented Jul 28, 2021

The ReadWrite tests were hanging in the past because of the thread pool issue. I'm not sure if the workaround we put it is sufficient with Mono.
Proper fix should be in #56346 (unless Mono needs some more)
The PRs may or may not be in sync.
cc: @kouvel

@wfurt
Copy link
Member

wfurt commented Jul 28, 2021

btw the overlap with #55642 quite clear IMHO

@kouvel
Copy link
Member

kouvel commented Jul 28, 2021 •
edited
Loading

Proper fix should be in #56346 (unless Mono needs some more)

That fix should cover Mono as well for the thread pool issue

@karelz
Copy link
Member Author

karelz commented Jul 28, 2021

OK, closing as fixed by #56346 (on 7/27)

I will check the failure rate in a week or two.

@karelz karelz closed this as completed Jul 28, 2021
HTTP/3 automation moved this from To Do (Low Priority) to Done Jul 28, 2021
@karelz karelz added the test-run-core Test failures in .NET Core test runs label Jul 31, 2021
@ghost ghost locked as resolved and limited conversation to collaborators Aug 30, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
area-System.Net.Quic test-run-core Test failures in .NET Core test runs
Projects
No open projects
HTTP/3
  
Done
Milestone
6.0.0
Development

No branches or pull requests
3 participants
@karelz @kouvel @wfurt