Http Stress Status Report #42211

ManickaP · 2020-09-14T18:17:06Z

Http Stress Status Report

What we've run so far:

OS	HTTP 1.1	HTTP 2.0	Notes
Windows	4+ h (Mana) -> 2 errors	12 h (JJ) -> 11 errors
	6 h (Miha) -> 8 errors	6 h (Miha) -> 7 errors
	10 h (Miha) -> 53 errors	7 h (Miha) -> 139 errors	7 h = 1 + 6 h
Linux	17 h (JJ) -> 0 errors	12 h (JJ) -> 608 errors	we should rerun as this may be environmental problem
	12 h (Furtik) -> 0 errors	12 h (Furtik) -> 104 errors	ran on released runtime (not on master)

HTTP 2.0 Error Statistics

Error Type	Linux	Windows
Success	135,727,755	105,273,552
Errors	712	157
System.Threading.Tasks.TaskCanceledException: The operation was canceled.	37	8
System.Threading.Tasks.TaskCanceledException: A task was canceled.	13	4
System.IO.IOException: The response ended prematurely while waiting for the next frame from the server.	464	18
System.Net.Sockets.SocketException (32): Broken pipe	198	0
System.Net.Sockets.SocketException (10054): An existing connection was forcibly closed by the remote host.	0	118
System.Net.Sockets.SocketException (10053): An established connection was aborted by the software in your host machine.	0	9

HTTP 1.1 Error Statistics

Error Type	Linux	Windows
Success	171,673,491	128,658,822
Errors	0	61
System.Net.Sockets.SocketException (10061): No connection could be made because the target machine actively refused it.	0	61

What we need to run:

More HTTP 1.1 Linux runs to confirm that we're clear. (easy, hi pri)
More HTTP 2.0 Linux runs to confirm that we have all error types captured. (easy, hi pri)
HTTP 2.0 tests against HTTPSys to eliminate/confirm server as the culprit. (mid, mid pri)
Run the matrix against 3.1 and compare. (hard, mid pri)

Existing issues, root caused:

HTTP 1.1 connect cancellation deadlock: Cancellation of Socket.ConnectAsync intermitently hangs #42198
HTTP 2.0 sending RESET before END_STREAM: HTTP/2 stress server intermitently fails in duplex scenarios #42200
HTTP 2.0 throwing TaskCancelledException as a reaction on GO_AWAY: HTTP/2 stress test TaskCanceledException when client hasn't cancelled #42472
NEW HTTP 2.0 infinite loop when connecting H2C to HTTP 1.0 localhost server: H2C connection to HTTP.sys on localhost causes infinite loop #42259

Discovered exceptions, not-investigated:

HTTP 2.0 System.Net.Sockets.SocketException (32): Broken pipe. (Linux only)
HTTP 2.0 System.IO.IOException: The response ended prematurely while waiting for the next frame from the server.
~~HTTP 2.0 System.Threading.Tasks.TaskCanceledException: The operation was canceled.~~ HTTP/2 stress test TaskCanceledException when client hasn't cancelled #42472
HTTP 2.0 System.Threading.Tasks.TaskCanceledException: A task was canceled. --> might be covered by HTTP/2 stress test TaskCanceledException when client hasn't cancelled #42472
HTTP 2.0 System.Net.Sockets.SocketException (10054): An existing connection was forcibly closed by the remote host. (Windows only)
HTTP 2.0 System.Net.Sockets.SocketException (10053): An established connection was aborted by the software in your host machine. (Windows only)
NEW HTTP 1.1 System.Net.Sockets.SocketException (10060): A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond. (Windows only)

The discovered exceptions confirm what we've collected so far from the pipelines: #40388.

Distributable tasks by priority:

More HTTP 1.1 Linux runs: http11run
More HTTP 2.0 Linux runs: http20run
Investigate Windows HTTP 1.1 errors: winErr3
Investigate Windows HTTP 2.0 errors: winErr1, winErr2
Provide fix for #42200
Provide fix for #42198
Help with HTTPSys client connection errors: httpSys: put on back-burner
Run the tests against .NET Core 3.1: net31: put on back-burner

Tips and Tricks for investigations:

Clear up docker containers and images after a product code change (docker container prune && docker image prune -a)
Once container is built, switch -b might be omitted for subsequent re-runs (skips the runtime build)
Don't use containers for investigations, they're slow and rebuild takes too long
- To run the app against locally built runtime, swap artifacts/bin/testhost/net5.0-Linux-Debug-x64/shared/Microsoft.NETCore.App/6.0.0/ with the globally installed runtime (/usr/share/dotnet/shared/Microsoft.NETCore.App/your-latest-5.0-version)
  - Make a backup of the global runtime!
  - Using testhost's corerun didn't work for me since the app depends on ASP .NET Core SDK
  - If you change the product code, rebuild just System.Net.Http and copy System.Net.Http.dll to the global runtime again
- Build System.Net.Http/tests/StressTests/HttpStress
- Open 2 terminals and run:
  - server: dotnet run -runMode server -aspnetlog
    - -aspnetlog: console logging of server errors
    - -serverUri https://localhost:5002: bind to a different port (when running multiple tests in parallel)
  - client: dotnet run -runMode client
    - -serverUri https://localhost:5002: connect to a different port (when running multiple tests in parallel)
    - -ops 1 2 3: run only operation 1, 2 and 3 (GET, PUT Slow, etc...)
    - -trace: saves internal client/server traces in a log file, very verbose, useable only for very short runs
  - more options in source code: https://github.com/dotnet/runtime/blob/master/src/libraries/System.Net.Http/tests/StressTests/HttpStress/Program.cs#L37-L62

If you have any improvements to the stress app or the containers, please create a PR and don't keep it just for yourself.
If you have more tips and tricks for running the tests, please share them.

The text was updated successfully, but these errors were encountered:

ghost · 2020-09-14T18:17:08Z

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

Fixes: stress client double read of content fixed fixed stress client hangs at start and stop leveraged HttpVersionPolicy increased pipeline timeout since we doubled the runs fixed base docker images to avoid missing IO.Pipelines Kestrel exception. Re-hauled tracing: added server file logging added log file rotation Minor renames. Contributes to: #42211 and #42198

antonfirsov · 2021-03-22T16:59:28Z

The description of #49999 has some useful links to stack traces according how failures occur today. The "new" HTTP 1.1 connection timeout seems to be a dominant issue on CI, failing on almost every Windows run.

antonfirsov · 2021-05-10T11:18:56Z

In the last 20 CI runs, there were 3 occurrences of the "System.Net.Sockets.SocketException (10054)" Windows + HTTP 2.0 case. I won't be surprised, if it's also a firewall issue being eventually solved by #52381. If not we should look into it.

We decided that we need to rule out the Windows Firewall as a cause for further failure types documented in #42211.

antonfirsov · 2021-06-28T10:36:02Z

I browsed through the failures of the last 25 days in the CI, and it seems that we are facing the same errors as las September. The only thing worth mentioning is that I've noticed a TaskCanceledException failure with a stack trace that comes from PUT Slow, instead of POST ExpectContinue I can't decide if it's covered by #42472 or not, we may want to check it:

System.Threading.Tasks.TaskCanceledException: The request was canceled due to the configured HttpClient.Timeout of 60 seconds elapsing.
-> System.TimeoutException: A task was canceled.
-> System.Threading.Tasks.TaskCanceledException: A task was canceled.
at System.Net.Http.Http2Connection.SendStreamDataAsync(Int32 streamId, ReadOnlyMemory`1 buffer, Boolean finalFlush, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http2Connection.cs:line 1482
at System.Net.Http.Http2Connection.Http2Stream.SendDataAsync(ReadOnlyMemory`1 buffer, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http2Stream.cs:line 1275
at HttpStress.ClientOperations.ByteAtATimeNoLengthContent.SerializeToStreamAsync(Stream stream, TransportContext context) in C:\app\ClientOperations.cs:line 623
at System.Net.Http.Http2Connection.Http2Stream.SendRequestBodyAsync(CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http2Stream.cs:line 291
at System.Net.Http.Http2Connection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http2Connection.cs:line 1941
at System.Net.Http.HttpConnectionPool.TrySendUsingHttp2Async(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs:line 833
at System.Net.Http.HttpConnectionPool.DetermineVersionAndSendAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs:line 897
at System.Net.Http.HttpConnectionPool.SendAndProcessAltSvcAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs:line 910
at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs:line 920
at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/RedirectHandler.cs:line 30
at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/HttpClient.cs:line 548

Full CI run

karelz · 2021-07-29T08:03:13Z

@antonfirsov can you please add note about your analysis from last week?

antonfirsov · 2021-07-29T14:40:36Z

Report 7/29

with manual observations after a brief look through the CI output of between 7/13-7/29, excluding the HTTP3 failures:

There is a new failure type now tracked under Regression in HTTP/1.1 stress tests: ObjectDisposedException ('SslStream') #56159
Other failures I've seen are known and described in the opening post
There are 2 cases where the Job run is unfinished. Likely an AZP anomaly, but we should keep monitoring if it happens again

Going back a bit more I see signs of #55261 (see here), but it doesn't seem to happen since 7/8 (EDIT: Because the test was disabled on 7/8).

antonfirsov · 2021-08-05T15:14:19Z

Report 8/5

brief look through the CI output between 7/29-8/5:

No new failure types in the test runs
Add SslApplicationProtocol.Http3 #56775 broke the last build, which is being fixed in Fix HttpStress build #56912

antonfirsov · 2021-08-13T09:51:06Z

Report 8/13

No new failure types in HTTP 1.1 & 2.0 in the last 1 week.

antonfirsov · 2021-08-24T15:19:42Z

Report 8/24

No new failure types in HTTP 1.1 & 2.0 since last report
There is one ~~occurrence~~ occurence of an unexpected ObjectDisposedException that happened after merging Do not drain HttpContentReadStream if the connection is disposed #57287. I'm reluctant to reopen Regression in HTTP/1.1 stress tests: ObjectDisposedException ('SslStream') #56159 right away, will do local runs and keep watching the CI, and open an issue if it happens again.

ManickaP · 2021-08-24T15:35:44Z

The occurrence is on 13.8., isn't that before the fix got merged?

antonfirsov · 2021-08-24T16:06:07Z

Sorry, linked wrong build, this one happened on the 18th:
https://dev.azure.com/dnceng/public/_build/results?buildId=1301334&view=logs&j=0da5d1d9-276d-5173-c4c4-9d4d4ed14fdb&t=45b31e99-b18c-506f-46cf-7d3a4d5140a9

karelz · 2021-08-25T12:15:23Z

@Vijay-Nirmal please file a separate issue with additional information. This issue is tracking general stress test runs, which may or may not be related to what you see. It is fine to link this issue. Thanks!

antonfirsov · 2023-10-12T14:48:32Z

Report 2023-10-12

`main` (9.0) branch

Looking at 3-4 runs against main since the fix of #93135.

HttpStress on `main`

No failures
Last run: 20231011.2

SslStress on `main`

No failures
Last run: 20231012.3

`release/8.0` branch

Looking at runs against release/8.0 since the last report (3 weeks).

HttpStress on `release/8.0`

1 x connection timeout at startup
1 x QuicException: Stream aborted by peer (258) similar to the case noticed in March: Http Stress Status Report #42211 (comment)

System.Net.Http.HttpRequestException: Error while copying content to a stream.
 ---> System.Net.Quic.QuicException: Stream aborted by peer (258).
   at System.Net.Quic.ResettableValueTaskSource.TryComplete(Exception exception, Boolean final) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/Internal/ResettableValueTaskSource.cs:line 169
   at System.Net.Quic.QuicStream.HandleEventPeerReceiveAborted(_PEER_RECEIVE_ABORTED_e__Struct& data) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/QuicStream.cs:line 559
   at System.Net.Quic.QuicStream.HandleStreamEvent(QUIC_STREAM_EVENT& streamEvent) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/QuicStream.cs:line 609
   at System.Net.Quic.QuicStream.NativeCallback(QUIC_HANDLE* connection, Void* context, QUIC_STREAM_EVENT* streamEvent) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/QuicStream.cs:line 640
--- End of stack trace from previous location ---
   at System.Net.Quic.ResettableValueTaskSource.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/Internal/ResettableValueTaskSource.cs:line 277
   at System.Net.Http.Http3RequestStream.WriteRequestContentAsync(ReadOnlyMemory`1 buffer, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 481
   at HttpStress.ClientOperations.ByteAtATimeNoLengthContent.SerializeToStreamAsync(Stream stream, TransportContext context) in C:\app\ClientOperations.cs:line 615
   at System.Net.Http.HttpContent.<CopyToAsync>g__WaitAsync|56_0(ValueTask copyTask) in /_/src/libraries/System.Net.Http/src/System/Net/Http/HttpContent.cs:line 399
   --- End of inner exception stack trace ---
   at System.Net.Http.HttpContent.<CopyToAsync>g__WaitAsync|56_0(ValueTask copyTask) in /_/src/libraries/System.Net.Http/src/System/Net/Http/HttpContent.cs:line 403
   at System.Net.Http.Http3RequestStream.SendContentAsync(HttpContent content, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 401
   at System.Net.Http.Http3RequestStream.SendAsync(CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 186
   at System.Net.Http.Http3RequestStream.SendAsync(CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 322
   at System.Net.Http.Http3Connection.SendAsync(HttpRequestMessage request, Int64 queueStartingTimestamp, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3Connection.cs:line 234
   at System.Net.Http.Http3Connection.SendAsync(HttpRequestMessage request, Int64 queueStartingTimestamp, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3Connection.cs:line 246
   at System.Net.Http.HttpConnectionPool.TrySendUsingHttp3Async(HttpRequestMessage request, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs:line 999
   at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs:line 1053
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/RedirectHandler.cs:line 30
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/HttpClient.cs:line 530
   at HttpStress.RequestContext.SendAsync(HttpRequestMessage request, HttpCompletionOption httpCompletion, Nullable`1 token) in C:\app\ClientOperations.cs:line 95
   at HttpStress.ClientOperations.<>c.<<get_Operations>b__1_13>d.MoveNext() in C:\app\ClientOperations.cs:line 453
--- End of stack trace from previous location ---
   at HttpStress.StressClient.<>c__DisplayClass17_0.<<StartCore>g__RunWorker|0>d.MoveNext() in C:\app\StressClient.cs:line 204

2 x 'H3_FRAME_ERROR' in a single run. We haven't seen this before. We should keep monitoring if it happens again.

System.Net.Http.HttpRequestException: An error occurred while sending the request.
 ---> System.Net.Http.HttpProtocolException: The HTTP/3 server sent invalid data on the connection. HTTP/3 error code 'H3_FRAME_ERROR' (0x106). (HttpProtocolError)
   at System.Net.Http.Http3RequestStream.SkipUnknownPayloadAsync(Int64 payloadLength, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 1092
   at System.Net.Http.Http3RequestStream.ReadFrameEnvelopeAsync(CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 860
   at System.Net.Http.Http3RequestStream.ReadNextDataFrameAsync(HttpResponseMessage response, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 1296
   at System.Net.Http.Http3RequestStream.ReadResponseContentAsync(HttpResponseMessage response, Memory`1 buffer, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 1187
   at System.Net.Http.Http3RequestStream.HandleReadResponseContentException(Exception ex, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 1270
   at System.Net.Http.Http3RequestStream.ReadResponseContentAsync(HttpResponseMessage response, Memory`1 buffer, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 1240
   at System.IO.Stream.<CopyToAsync>g__Core|27_0(Stream source, Stream destination, Int32 bufferSize, CancellationToken cancellationToken) in /_/src/libraries/System.Private.CoreLib/src/System/IO/Stream.cs:line 108
   at System.Net.Http.HttpConnectionResponseContent.<SerializeToStreamAsync>g__Impl|6_0(Stream stream, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionResponseContent.cs:line 61
   at System.Net.Http.HttpContent.LoadIntoBufferAsyncCore(Task serializeToStreamTask, MemoryStream tempBuffer) in /_/src/libraries/System.Net.Http/src/System/Net/Http/HttpContent.cs:line 515
   --- End of inner exception stack trace ---
   at System.Net.Http.Http3RequestStream.SendAsync(CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 281
   at System.Net.Http.Http3RequestStream.SendAsync(CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 322
   at System.Net.Http.Http3Connection.SendAsync(HttpRequestMessage request, Int64 queueStartingTimestamp, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3Connection.cs:line 234
   at System.Net.Http.Http3Connection.SendAsync(HttpRequestMessage request, Int64 queueStartingTimestamp, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3Connection.cs:line 246
   at System.Net.Http.HttpConnectionPool.TrySendUsingHttp3Async(HttpRequestMessage request, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs:line 999
   at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs:line 1053
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/RedirectHandler.cs:line 30
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/HttpClient.cs:line 530
   at HttpStress.RequestContext.SendAsync(HttpRequestMessage request, HttpCompletionOption httpCompletion, Nullable`1 token) in C:\app\ClientOperations.cs:line 95
   at HttpStress.ClientOperations.<>c.<<get_Operations>b__1_13>d.MoveNext() in C:\app\ClientOperations.cs:line 453

Last run: 20231010.2

SslStress on `release/8.0`

No failures
Last run: 20231012.1

karelz · 2023-10-19T13:17:46Z

Report 2023-10-19

`main` (9.0) branch

HttpStress on `main`

7x runs since last report - 5x failures
Last run: 20231018.2

Failures:

4x HttpStress, HTTP/3: content mismatch in POST Duplex Slow #76183
- 1x in run 20231013.2
- 3x in run 20231015.2
1x QuicException: Connection aborted by peer - same as in release/8.0 report on 2023/10/12

System.Net.Http.HttpRequestException: Error while copying content to a stream.
  ---> System.Net.Quic.QuicException: Connection aborted by peer (261).
    at System.Net.Quic.ResettableValueTaskSource.TryComplete(Exception exception, Boolean final) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/Internal/ResettableValueTaskSource.cs:line 180
    at System.Net.Quic.QuicStream.HandleEventShutdownComplete(_SHUTDOWN_COMPLETE_e__Struct& data) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/QuicStream.cs:line 614
    at System.Net.Quic.QuicStream.HandleStreamEvent(QUIC_STREAM_EVENT& streamEvent) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/QuicStream.cs:line 636
    at System.Net.Quic.QuicStream.NativeCallback(QUIC_HANDLE* stream, Void* context, QUIC_STREAM_EVENT* streamEvent) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/QuicStream.cs:line 665
 --- End of stack trace from previous location ---
    at System.Net.Quic.ResettableValueTaskSource.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/Internal/ResettableValueTaskSource.cs:line 256
    at System.Net.Quic.QuicStream.ReadAsync(Memory`1 buffer, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/QuicStream.cs:line 307
    at System.Net.Http.Http3RequestStream.ReadFrameEnvelopeAsync(CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 815
    at System.Net.Http.Http3RequestStream.ReadResponseAsync(CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 339
    --- End of inner exception stack trace ---
    at System.Net.Http.HttpContent.<CopyToAsync>g__WaitAsync|56_0(ValueTask copyTask) in /_/src/libraries/System.Net.Http/src/System/Net/Http/HttpContent.cs:line 403
    at System.Net.Http.Http3RequestStream.SendContentAsync(HttpContent content, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 401
    at System.Net.Http.Http3RequestStream.SendAsync(CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 186
    at System.Net.Http.Http3RequestStream.SendAsync(CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 322
    at System.Net.Http.Http3Connection.SendAsync(HttpRequestMessage request, Int64 queueStartingTimestamp, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3Connection.cs:line 236
    at System.Net.Http.Http3Connection.SendAsync(HttpRequestMessage request, Int64 queueStartingTimestamp, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3Connection.cs:line 248
    at System.Net.Http.HttpConnectionPool.TrySendUsingHttp3Async(HttpRequestMessage request, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs:line 999
    at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs:line 1053
    at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/RedirectHandler.cs:line 30
    at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/HttpClient.cs:line 530
    at HttpStress.RequestContext.SendAsync(HttpRequestMessage request, HttpCompletionOption httpCompletion, Nullable`1 token) in C:\app\ClientOperations.cs:line 95
    at HttpStress.ClientOperations.<>c.<<get_Operations>b__1_13>d.MoveNext() in C:\app\ClientOperations.cs:line 453
 --- End of stack trace from previous location ---
    at HttpStress.StressClient.<>c__DisplayClass17_0.<<StartCore>g__RunWorker|0>d.MoveNext() in C:\app\StressClient.cs:line 204

SslStress on `main`

6x runs since last report - 3x failures
Last run: 20231018.1

Failures:

3x [SslStress] SocketException (111): Connection refused #94222 (System.Net.Sockets.SocketException (111): Connection refused) - same as in report on 2022/9/14 (1 year old)
- 2x in run 20231014.1
- 1x in run 20231018.1

20231014.1 - 1st failure:

System.Net.Sockets.SocketException (111): Connection refused
    at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/Socket.Tasks.cs:line 1378
    at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token) in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/Socket.Tasks.cs:line 1294
    at System.Threading.Tasks.ValueTask.ValueTaskSourceAsTask.<>c.<.cctor>b__4_0(Object state) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/ValueTask.cs:line 258
 --- End of stack trace from previous location ---
    at System.Net.Sockets.TcpClient.CompleteConnectAsync(Task task) in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/TCPClient.cs:line 193
    at SslStress.SslClientBase.<StartCore>g__RunSingleWorker|15_1(Int32 workerId, Random random) in /app/System.Net.Security/tests/StressTests/SslStress/SslClientBase.cs:line 116

20231014.1 - 2nd failure

System.Net.Sockets.SocketException (111): Connection refused
    at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.CreateException(SocketError error, Boolean forAsyncThrow) in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/Socket.Tasks.cs:line 1387
    at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ConnectAsync(Socket socket) in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/Socket.Tasks.cs:line 1224
    at System.Net.Sockets.Socket.ConnectAsync(EndPoint remoteEP, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/Socket.Tasks.cs:line 100
    at System.Net.Sockets.Socket.ConnectAsync(EndPoint remoteEP) in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/Socket.Tasks.cs:line 78
    at System.Net.Sockets.Socket.ConnectAsync(IPAddress address, Int32 port) in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/Socket.Tasks.cs:line 135
    at System.Net.Sockets.TcpClient.ConnectAsync(IPAddress address, Int32 port) in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/TCPClient.cs:line 175
    at SslStress.SslClientBase.<StartCore>g__RunSingleWorker|15_1(Int32 workerId, Random random) in /app/System.Net.Security/tests/StressTests/SslStress/SslClientBase.cs:line 116
    at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecutionContextCallback(Object s) in /_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncTaskMethodBuilderT.cs:line 288
    at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread) in /_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncTaskMethodBuilderT.cs:line 368
    at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext() in /_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncTaskMethodBuilderT.cs:line 346
    at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/TaskContinuation.cs:line 795
    at System.Threading.Tasks.Task.RunContinuations(Object continuationObject) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/Task.cs:line 3456
    at System.Threading.Tasks.Task.FinishSlow(Boolean userDelegateExecute) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/Task.cs:line 1999
    at System.Threading.Tasks.Task.TrySetException(Object exceptionObject) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/Task.cs:line 3371
    at System.Threading.Tasks.ValueTask.ValueTaskSourceAsTask.<>c.<.cctor>b__4_0(Object state) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/ValueTask.cs:line 258
    at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.OnCompleted(SocketAsyncEventArgs _) in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/Socket.Tasks.cs:line 1031
    at System.Net.Sockets.SocketAsyncEventArgs.OnCompletedInternal() in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/SocketAsyncEventArgs.cs:line 210
    at System.Net.Sockets.SocketAsyncEventArgs.FinishOperationAsyncFailure(SocketError socketError, Int32 bytesTransferred, SocketFlags flags) in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/SocketAsyncEventArgs.cs:line 636
    at System.Net.Sockets.SocketAsyncEventArgs.CompletionCallback(Int32 bytesTransferred, SocketFlags flags, SocketError socketError) in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/SocketAsyncEventArgs.Unix.cs:line 391
    at System.Net.Sockets.SocketAsyncEventArgs.ConnectCompletionCallback(SocketError socketError) in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/SocketAsyncEventArgs.Unix.cs:line 72
    at System.Net.Sockets.SocketAsyncContext.ConnectOperation.InvokeCallback(Boolean allowPooling) in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/SocketAsyncContext.Unix.cs:line 665
    at System.Net.Sockets.SocketAsyncContext.OperationQueue`1.ProcessAsyncOperation(TOperation op) in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/SocketAsyncContext.Unix.cs:line 1004
    at System.Net.Sockets.SocketAsyncContext.ProcessAsyncWriteOperation(WriteOperation op) in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/SocketAsyncContext.Unix.cs:line 1415
    at System.Net.Sockets.SocketAsyncContext.WriteOperation.System.Threading.IThreadPoolWorkItem.Execute() in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/SocketAsyncContext.Unix.cs:line 347
    at System.Net.Sockets.SocketAsyncContext.AsyncOperation.Process() in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/SocketAsyncContext.Unix.cs:line 291
    at System.Net.Sockets.SocketAsyncContext.HandleEvents(SocketEvents events) in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/SocketAsyncContext.Unix.cs:line 2202
    at System.Net.Sockets.SocketAsyncEngine.System.Threading.IThreadPoolWorkItem.Execute() in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/SocketAsyncEngine.Unix.cs:line 242
    at System.Threading.ThreadPoolWorkQueue.Dispatch() in /_/src/libraries/System.Private.CoreLib/src/System/Threading/ThreadPoolWorkQueue.cs:line 913
    at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart() in /_/src/libraries/System.Private.CoreLib/src/System/Threading/PortableThreadPool.WorkerThread.NonBrowser.cs:line 102
 --- End of stack trace from previous location ---
    at System.Net.Sockets.TcpClient.CompleteConnectAsync(Task task) in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/TCPClient.cs:line 193
    at SslStress.SslClientBase.<StartCore>g__RunSingleWorker|15_1(Int32 workerId, Random random) in /app/System.Net.Security/tests/StressTests/SslStress/SslClientBase.cs:line 116

20231018.1

System.Net.Sockets.SocketException (111): Connection refused
    at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/Socket.Tasks.cs:line 1378
    at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token) in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/Socket.Tasks.cs:line 1294
    at System.Threading.Tasks.ValueTask.ValueTaskSourceAsTask.<>c.<.cctor>b__4_0(Object state) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/ValueTask.cs:line 258
 --- End of stack trace from previous location ---
    at System.Net.Sockets.TcpClient.CompleteConnectAsync(Task task) in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/TCPClient.cs:line 193
    at SslStress.SslClientBase.<StartCore>g__RunSingleWorker|15_1(Int32 workerId, Random random) in /app/System.Net.Security/tests/StressTests/SslStress/SslClientBase.cs:line 116

`release/8.0` branch

HttpStress on `release/8.0`

6x runs since last report - 2x failures
Last run: 20231018.3

Failures:

1x HttpStress, HTTP/3: content mismatch in POST Duplex Slow #76183 - in 20231018.3

System.Exception: Expected response content "4OUyhzcbKT32orLXUBOGP9qVKs42TcOum5uX54YAX6I5otXzGki9hGlvXydpvZ7h49FixoRQcEjJqlgRK4mjuFCWJz06QjEyyb0edp2SIHXSXS4lICVqmNnpwjOxoKzs0kLFl1oU2KRchkzruEoEENBanAiuQ4dxHeSm6pzGN0oRwA8CjX3ZWlecZ", got "4UyhzcbKT32orLXUBOGP9qVKs42TcOum5uX54YAX6I5otXzGki9hGlvXydpvZ7h49FixoRQcEjJqlgRK4mjuFCWJz06QjEyyb0edp2SIHXSXS4lICVqmNnpwjOxoKzs0kLFl1oU2KRchkzruEoEENBanAiuQ4dxHeSm6pzGN0oRwA8CjX3ZWlecZ".
  Diverging at index 1. server checksum does not match client value.
    at HttpStress.ClientOperations.ValidateContent(String expectedContent, String actualContent, String details) in /app/ClientOperations.cs:line 494
    at HttpStress.ClientOperations.<>c.<<get_Operations>b__1_8>d.MoveNext()
 --- End of stack trace from previous location ---
    at HttpStress.StressClient.<>c__DisplayClass17_0.<<StartCore>g__RunWorker|0>d.MoveNext() in /app/StressClient.cs:line 204

1x Docker NanoServer failure in 20231014.3

Step 8/15 : RUN Invoke-WebRequest -Uri https://dot.net/v1/dotnet-install.ps1 -OutFile .\dotnet-install.ps1
 ---> Running in 5abbed8be7c4
Invoke-WebRequest: The service is unavailable.
The command 'pwsh -Command $ErrorActionPreference = 'Stop'; $ProgressPreference = 'SilentlyContinue'; Invoke-WebRequest -Uri https://dot.net/v1/dotnet-install.ps1 -OutFile .\dotnet-install.ps1' returned a non-zero code: 1
##[error]PowerShell exited with code '1'.

SslStress on `release/8.0`

4x runs since last report - No failures
Last run: 20231018.2

antonfirsov · 2023-10-31T16:16:04Z

Report 2023-10-31

`main` (9.0) branch

HttpStress on `main`

3 x HttpStress, HTTP/3: content mismatch in POST Duplex Slow #76183
1 x [HTTP/3] H3_FRAME_UNEXPECTED in stress tests #87559 (reopened)
last run: 20231031.3

SslStress on `main`

5 x [SslStress] SocketException (111): Connection refused #94222 (Also happenned 4x in the previous report, so opened an issue)
Last run: 20231031.2

`release/8.0` branch

HttpStress on `release/8.0`

1 x QuicException: Stream aborted by peer (267) similar to Http Stress Status Report #42211 (comment)

httpstress-client-1  | System.Net.Http.HttpRequestException: Error while copying content to a stream.
httpstress-client-1  |  ---> System.Net.Quic.QuicException: Stream aborted by peer (267).
httpstress-client-1  |    at System.Net.Quic.ResettableValueTaskSource.TryComplete(Exception exception, Boolean final) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/Internal/ResettableValueTaskSource.cs:line 169
httpstress-client-1  |    at System.Net.Quic.QuicStream.HandleEventPeerReceiveAborted(_PEER_RECEIVE_ABORTED_e__Struct& data) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/QuicStream.cs:line 559
httpstress-client-1  |    at System.Net.Quic.QuicStream.HandleStreamEvent(QUIC_STREAM_EVENT& streamEvent) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/QuicStream.cs:line 609
httpstress-client-1  |    at System.Net.Quic.QuicStream.NativeCallback(QUIC_HANDLE* connection, Void* context, QUIC_STREAM_EVENT* streamEvent) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/QuicStream.cs:line 640
httpstress-client-1  | --- End of stack trace from previous location ---
httpstress-client-1  |    at System.Net.Quic.ResettableValueTaskSource.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/Internal/ResettableValueTaskSource.cs:line 277
httpstress-client-1  |    at System.Net.Http.Http3RequestStream.WriteRequestContentAsync(ReadOnlyMemory`1 buffer, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 481
httpstress-client-1  |    at HttpStress.ClientOperations.ByteAtATimeNoLengthContent.SerializeToStreamAsync(Stream stream, TransportContext context) in C:\app\ClientOperations.cs:line 615
httpstress-client-1  |    at System.Net.Http.HttpContent.<CopyToAsync>g__WaitAsync|56_0(ValueTask copyTask) in /_/src/libraries/System.Net.Http/src/System/Net/Http/HttpContent.cs:line 399
httpstress-client-1  |    --- End of inner exception stack trace ---
httpstress-client-1  |    at System.Net.Http.HttpContent.<CopyToAsync>g__WaitAsync|56_0(ValueTask copyTask) in /_/src/libraries/System.Net.Http/src/System/Net/Http/HttpContent.cs:line 403
httpstress-client-1  |    at System.Net.Http.Http3RequestStream.SendContentAsync(HttpContent content, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 401
httpstress-client-1  |    at System.Net.Http.Http3RequestStream.SendAsync(CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 186
httpstress-client-1  |    at System.Net.Http.Http3RequestStream.SendAsync(CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 322
httpstress-client-1  |    at System.Net.Http.Http3Connection.SendAsync(HttpRequestMessage request, Int64 queueStartingTimestamp, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3Connection.cs:line 234
httpstress-client-1  |    at System.Net.Http.Http3Connection.SendAsync(HttpRequestMessage request, Int64 queueStartingTimestamp, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3Connection.cs:line 246
httpstress-client-1  |    at System.Net.Http.HttpConnectionPool.TrySendUsingHttp3Async(HttpRequestMessage request, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs:line 999
httpstress-client-1  |    at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs:line 1053

Last run: 20231031.2

SslStress on `release/8.0`

No failures
Last run: 20231031.1

antonfirsov · 2023-11-09T17:02:35Z

Report 2023-11-09

`main` (9.0) branch

HttpStress on `main`

 System.Net.Http.HttpRequestException: Error while copying content to a stream.
  ---> System.Net.Http.HttpProtocolException: The HTTP/3 server reset the stream. HTTP/3 error code 'H3_REQUEST_REJECTED' (0x10b). (HttpProtocolError)
  ---> System.Net.Quic.QuicException: Stream aborted by peer (267).
    at System.Net.Quic.ResettableValueTaskSource.TryComplete(Exception exception, Boolean final) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/Internal/ResettableValueTaskSource.cs:line 180
    at System.Net.Quic.QuicStream.HandleEventPeerSendAborted(_PEER_SEND_ABORTED_e__Struct& data) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/QuicStream.cs:line 579
    at System.Net.Quic.QuicStream.HandleStreamEvent(QUIC_STREAM_EVENT& streamEvent) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/QuicStream.cs:line 633
    at System.Net.Quic.QuicStream.NativeCallback(QUIC_HANDLE* stream, Void* context, QUIC_STREAM_EVENT* streamEvent) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/QuicStream.cs:line 665
 --- End of stack trace from previous location ---
    at System.Net.Quic.ResettableValueTaskSource.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/Internal/ResettableValueTaskSource.cs:line 256
    at System.Net.Quic.QuicStream.ReadAsync(Memory`1 buffer, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/QuicStream.cs:line 307
    at System.Net.Http.Http3RequestStream.ReadFrameEnvelopeAsync(CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 815
    at System.Net.Http.Http3RequestStream.ReadNextDataFrameAsync(HttpResponseMessage response, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 1296
    at System.Net.Http.Http3RequestStream.ReadResponseContentAsync(HttpResponseMessage response, Memory`1 buffer, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 1187
    --- End of inner exception stack trace ---
    at System.Net.Http.Http3RequestStream.HandleReadResponseContentException(Exception ex, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 1259
    at System.Net.Http.Http3RequestStream.ReadResponseContentAsync(HttpResponseMessage response, Memory`1 buffer, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 1240
    at System.IO.Stream.<CopyToAsync>g__Core|27_0(Stream source, Stream destination, Int32 bufferSize, CancellationToken cancellationToken) in /_/src/libraries/System.Private.CoreLib/src/System/IO/Stream.cs:line 108
    at System.Net.Http.HttpConnectionResponseContent.<SerializeToStreamAsync>g__Impl|6_0(Stream stream, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionResponseContent.cs:line 61
    at System.Net.Http.HttpContent.LoadIntoBufferAsyncCore(Task serializeToStreamTask, MemoryStream tempBuffer) in /_/src/libraries/System.Net.Http/src/System/Net/Http/HttpContent.cs:line 515
    --- End of inner exception stack trace ---
    at System.Net.Http.HttpContent.LoadIntoBufferAsyncCore(Task serializeToStreamTask, MemoryStream tempBuffer) in /_/src/libraries/System.Net.Http/src/System/Net/Http/HttpContent.cs:line 521
    at System.Net.Http.HttpContent.WaitAndReturnAsync[TState,TResult](Task waitTask, TState state, Func`2 returnFunc) in /_/src/libraries/System.Net.Http/src/System/Net/Http/HttpContent.cs:line 833
    at HttpStress.ClientOperations.<>c.<<get_Operations>b__1_8>d.MoveNext() in C:\app\ClientOperations.cs:line 372
 --- End of stack trace from previous location ---
    at HttpStress.StressClient.<>c__DisplayClass17_0.<<StartCore>g__RunWorker|0>d.MoveNext() in C:\app\StressClient.cs:line 204

Last run: 20231109.3

SslStress on `main`

2 x [SslStress] SocketException (111): Connection refused #94222
Last run: 20231109.2

`release/8.0` branch

HttpStress on `release/8.0`

No failures
Last run: 20231109.2

SslStress on `release/8.0`

No failures
Last run: 20231109.1

antonfirsov · 2024-01-04T15:21:02Z

Report 2024-01-04

Looking Linux runs since 2023-12-05. Windows builds are broken across all branches, see #95750.

`main` (9.0) branch

HttpStress on `main`

2x HttpStress, HTTP/3: content mismatch in POST Duplex Slow #76183
1x Timeout at teardown. Likely an infrastructural issue, we should watch if it happens again.
Last run: 20240103.3

SslStress on `main`

14x [SslStress] SocketException (111): Connection refused #94222. This seems to be an issue affecting almost half of the runs again on main.
Last run: 20240104.2

`release/8.0` branch

HttpStress on `release/8.0`

No failures on Linux
Last run: 20240104.1

SslStress on `release/8.0`

No failures on Linux
Last run: 20231216.2

antonfirsov · 2024-01-18T16:30:57Z

Report 2024-01-18

Looking Linux runs since 2024-01-05. Windows builds are broken across all branches, see #95750.

`main` (9.0) branch

HttpStress on `main`

1 x crash, malloc(): unaligned tcache chunk detected

client_1  | [createdump] Gathering state for process 7 dotnet
client_1  | [createdump] Crashing thread 0014 signal 6 (0006)
client_1  | [createdump] Writing minidump with heap to file /dumps-share/coredump.7
client_1  | [createdump] Written 556867584 bytes (135954 pages) to core file
client_1  | [createdump] Target process is alive
client_1  | [createdump] Dump successfully written in 737ms

https://dev.azure.com/dnceng-public/public/_build/results?buildId=525074&view=logs&j=22fea640-1099-5f32-ec5d-316ad83f359a&t=a923b061-8445-5c16-cd6e-848990587cb0

Crash dump attached. We should look at it if it happens again.
DumpsLinux.zip

Time out at shutdown (##[error]The task has timed out):
https://dev.azure.com/dnceng-public/public/_build/results?buildId=527377&view=logs&j=22fea640-1099-5f32-ec5d-316ad83f359a&t=a923b061-8445-5c16-cd6e-848990587cb0
Last run: 20240118.3

SslStress on `main`

11x [SslStress] SocketException (111): Connection refused #94222
Last run: 20240118.3

`release/8.0` branch

HttpStress on `release/8.0`

No failures on Linux
Last run: #20240118.2

SslStress on `release/8.0`

No failures on Linux
Last run: 20240118.2

antonfirsov · 2024-02-15T16:25:17Z

Report 2024-02-15

Looking Linux runs since 2024-01-18. Windows builds are broken across all branches, see #95750.

`main` (9.0) branch

HttpStress on `main`

1 x crash, malloc(): mismatching next->prev_size (unsorted) in the HTTP/3.0 run: https://dev.azure.com/dnceng-public/public/_build/results?buildId=567734&view=logs&j=22fea640-1099-5f32-ec5d-316ad83f359a&t=a923b061-8445-5c16-cd6e-848990587cb0

client_1  | malloc(): mismatching next->prev_size (unsorted)
client_1  | Aborted (core dumped)
httpstress_client_1 exited with code 134

This looks very similar to the malloc crash detected in the previous run. Waiting for one more occurence to open an issue.

Crash dump:
DumpsLinux (1).zip

Last run: 20240215.2

SslStress on `main`

8 x [SslStress] SocketException (111): Connection refused #94222
Last run: 20240215.2

`release/8.0` branch

HttpStress on `release/8.0`

1 x HttpStress, HTTP/3: content mismatch in POST Duplex Slow #76183 (comment)
Last run: 20240215.1

SslStress on `release/8.0`

No failures on Linux
Last run: 20240215.1

antonfirsov · 2024-02-29T15:56:47Z

Report 2024-02-29

Looking Linux runs since 2024-01-15. Windows builds are broken across all branches, see #95750.

`main` (9.0) branch

HttpStress on `main`

1 x H3_REQUEST_INCOMPLETE with QuicException: Stream aborted by peer (269) on Linux:

https://dev.azure.com/dnceng-public/public/_build/results?buildId=580954&view=logs&j=22fea640-1099-5f32-ec5d-316ad83f359a&t=a923b061-8445-5c16-cd6e-848990587cb0

System.Net.Http.HttpRequestException: An error occurred while sending the request.
 ---> System.Net.Http.HttpProtocolException: The HTTP/3 server reset the stream. HTTP/3 error code 'H3_REQUEST_INCOMPLETE' (0x10d). (HttpProtocolError)
 ---> System.Net.Quic.QuicException: Stream aborted by peer (269).
   at System.Net.Quic.ResettableValueTaskSource.TryComplete(Exception exception, Boolean final) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/Internal/ResettableValueTaskSource.cs:line 177
   at System.Net.Quic.QuicStream.HandleEventPeerSendAborted(_PEER_SEND_ABORTED_e__Struct& data) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/QuicStream.cs:line 578
   at System.Net.Quic.QuicStream.HandleStreamEvent(QUIC_STREAM_EVENT& streamEvent) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/QuicStream.cs:line 633
   at System.Net.Quic.QuicStream.NativeCallback(QUIC_HANDLE* stream, Void* context, QUIC_STREAM_EVENT* streamEvent) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/QuicStream.cs:line 665
--- End of stack trace from previous location ---
   at System.Net.Quic.ResettableValueTaskSource.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/Internal/ResettableValueTaskSource.cs:line 251
   at System.Net.Quic.QuicStream.ReadAsync(Memory`1 buffer, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/QuicStream.cs:line 306
   at System.Net.Http.Http3RequestStream.ReadFrameEnvelopeAsync(CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 815
   at System.Net.Http.Http3RequestStream.ReadResponseAsync(CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 339
   at System.Net.Http.Http3RequestStream.SendAsync(CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 206
   --- End of inner exception stack trace ---
   --- End of inner exception stack trace ---
   at System.Net.Http.Http3RequestStream.SendAsync(CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 266
   at System.Net.Http.Http3RequestStream.SendAsync(CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 206
   at System.Net.Http.Http3RequestStream.SendAsync(CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 322
   at System.Net.Http.Http3Connection.SendAsync(HttpRequestMessage request, Int64 queueStartingTimestamp, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3Connection.cs:line 236
   at System.Net.Http.Http3Connection.SendAsync(HttpRequestMessage request, Int64 queueStartingTimestamp, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3Connection.cs:line 248
   at System.Net.Http.HttpConnectionPool.TrySendUsingHttp3Async(HttpRequestMessage request, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs:line 997

⬆️ this is a new error, we should open an issue if it happens again.

1 x HttpStress, HTTP/3: content mismatch in POST Duplex Slow #76183
Last run: 20240229.3

SslStress on `main`

No failures on Linux
Last run: 20240229.3

`release/8.0` branch

HttpStress on `release/8.0`

1 x HttpStress, HTTP/3: content mismatch in POST Duplex Slow #76183
Last run: 20240229.2

SslStress on `release/8.0`

No failures on Linux
Last run: 20240229.2

antonfirsov · 2024-03-19T16:01:57Z

Report 2024-03-19

Looking Linux runs since 2024-02-29. Windows builds are broken across all branches, see #95750.

`main` (9.0) branch

HttpStress on `main`

1 x [HttpStress] H3_REQUEST_INCOMPLETE on Linux #99959
1 x HttpStress, HTTP/3: content mismatch in POST Duplex Slow #76183 (comment)
1 x [HttpStress] malloc crashes in HTTP/3 runs #99960
Last run: 20240319.2

SslStress on `main`

4 x [SslStress] SocketException (111): Connection refused #94222
Last run: 20240319.3

`release/8.0` branch

HttpStress on `release/8.0`

1 x [HttpStress] H3_REQUEST_INCOMPLETE on Linux #99959
1 x [HTTP/3] H3_FRAME_UNEXPECTED in stress tests #87559
1 x HttpStress, HTTP/3: content mismatch in POST Duplex Slow #76183
Last run: 20240319.3

SslStress on `release/8.0`

No failures on Linux
Last run: 20240319.4

antonfirsov · 2024-03-26T16:29:34Z

Report 2024-03-26

Looking Linux runs since 2024-03-19. Windows runs are up on main since Sunday.

`main` (9.0) branch

HttpStress on `main`

1 x [HttpStress] malloc crashes in HTTP/3 runs #99960 (Linux)
1 x [HttpStress] H3_REQUEST_INCOMPLETE on Linux #99959 (Linux)
1 x HttpStress, HTTP/3: content mismatch in POST Duplex Slow #76183 (Windows)

SslStress on `main`

1 x [SslStress] SocketException (111): Connection refused #94222

`release/8.0` branch

Windows builds are still broken, #100145 needs a backport.

HttpStress on `release/8.0`

1 x [SslStress] SocketException (111): Connection refused #94222

SslStress on `release/8.0`

No failures on Linux

antonfirsov · 2024-04-02T17:06:59Z

Report 2024-04-02

Looking at all runs since 2024-03-26.

`main` (9.0) branch

HttpStress on `main`

200-300 hits of HttpStress, HTTP/3: content mismatch in POST Duplex Slow #76183 per run on Linux
30-50 hits of [HTTP/3] [HttpStress] QuicException: Stream aborted by peer (258) in (POST Duplex | PUT) Slow #100534 per run on Linux, 1 hit on Windows

SslStress on `main`

No failures

`release/8.0` branch

HttpStress on `release/8.0`

1 x HttpStress, HTTP/3: content mismatch in POST Duplex Slow #76183 on Linux

SslStress on `release/8.0`

1x SslStress: SocketException (995) WSA_OPERATION_ABORTED on Windows #76044. Waiting for more stats before reopening.

antonfirsov · 2024-04-09T14:46:38Z

Report 2024-04-09

Looking at runs since 2024-04-02.

`main` (9.0) branch

HttpStress on `main`

200-300 hits of HttpStress, HTTP/3: content mismatch in POST Duplex Slow #76183 per run on Linux
30-50 hits of [HTTP/3] [HttpStress] QuicException: Stream aborted by peer (258) in (POST Duplex | PUT) Slow #100534 per run on Linux

SslStress on `main`

No failures

`release/8.0` branch

HttpStress on `release/8.0`

2 x HttpStress, HTTP/3: content mismatch in POST Duplex Slow #76183 (1 on Windows 1 on Linux)

SslStress on `release/8.0`

No failures

antonfirsov · 2024-04-26T18:28:46Z

Report 2024-04-26

Looking at runs since 2024-04-09.

`main` (9.0) branch

HttpStress on `main`

The extreme amount of hits for #76183 became lower, and hits for #100534 ceased, however we are seeing a couple of failures similar to #76183 accumulating. Let's see if the frequency changes before opening separate issues.

13 x HttpStress, HTTP/3: content mismatch in POST Duplex Slow #76183, happens on both Linux and Windows
2 x [HttpStress] H3_REQUEST_INCOMPLETE on Linux #99959 H3_REQUEST_INCOMPLETE on Windows
2 x H3_FRAME_ERROR in POST Duplex Slow in a single run on Windows

https://dev.azure.com/dnceng-public/public/_build/results?buildId=642813&view=logs&j=0bc77094-9fcd-5c38-f6e4-27d2ae131589&t=4fd93899-d50b-5dc1-f9eb-66ab1b883c22

System.Net.Http.HttpRequestException: Error while copying content to a stream.
 ---> System.Net.Http.HttpProtocolException: The HTTP/3 server sent invalid data on the connection. HTTP/3 error code 'H3_FRAME_ERROR' (0x106). (HttpProtocolError)
   at System.Net.Http.Http3RequestStream.SkipUnknownPayloadAsync(Int64 payloadLength, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 1080
   at System.Net.Http.Http3RequestStream.ReadFrameEnvelopeAsync(CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 848
   at System.Net.Http.Http3RequestStream.ReadNextDataFrameAsync(HttpResponseMessage response, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 1284
   at System.Net.Http.Http3RequestStream.ReadResponseContentAsync(HttpResponseMessage response, Memory`1 buffer, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 1175
   at System.Net.Http.Http3RequestStream.HandleReadResponseContentException(Exception ex, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 1258
   at System.Net.Http.Http3RequestStream.ReadResponseContentAsync(HttpResponseMessage response, Memory`1 buffer, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 1228
   at System.IO.Stream.<CopyToAsync>g__Core|27_0(Stream source, Stream destination, Int32 bufferSize, CancellationToken cancellationToken) in /_/src/libraries/System.Private.CoreLib/src/System/IO/Stream.cs:line 108
   at System.Net.Http.HttpConnectionResponseContent.<SerializeToStreamAsync>g__Impl|6_0(Stream stream, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionResponseContent.cs:line 61
   at System.Net.Http.HttpContent.LoadIntoBufferAsyncCore(Task serializeToStreamTask, MemoryStream tempBuffer) in /_/src/libraries/System.Net.Http/src/System/Net/Http/HttpContent.cs:line 515
   --- End of inner exception stack trace ---
   at System.Net.Http.HttpContent.LoadIntoBufferAsyncCore(Task serializeToStreamTask, MemoryStream tempBuffer) in /_/src/libraries/System.Net.Http/src/System/Net/Http/HttpContent.cs:line 521
   at System.Net.Http.HttpContent.WaitAndReturnAsync[TState,TResult](Task waitTask, TState state, Func`2 returnFunc) in /_/src/libraries/System.Net.Http/src/System/Net/Http/HttpContent.cs:line 833
   at HttpStress.ClientOperations.<>c.<<get_Operations>b__1_8>d.MoveNext() in C:\app\ClientOperations.cs:line 372

1 x H3_REQUEST_REJECTED in GET Aborted on Windows, similarly to Http Stress Status Report #42211 (comment)

https://dev.azure.com/dnceng-public/public/_build/results?buildId=651163&view=logs&j=0bc77094-9fcd-5c38-f6e4-27d2ae131589&t=4fd93899-d50b-5dc1-f9eb-66ab1b883c22

System.Net.Http.HttpRequestException: Error while copying content to a stream.
 ---> System.Net.Http.HttpProtocolException: The HTTP/3 server reset the stream. HTTP/3 error code 'H3_REQUEST_REJECTED' (0x10b). (HttpProtocolError)
 ---> System.Net.Quic.QuicException: Stream aborted by peer (267).
   at System.Net.Quic.ResettableValueTaskSource.TryComplete(Exception exception, Boolean final) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/Internal/ResettableValueTaskSource.cs:line 177
   at System.Net.Quic.QuicStream.HandleEventPeerSendAborted(_PEER_SEND_ABORTED_e__Struct& data) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/QuicStream.cs:line 580
   at System.Net.Quic.QuicStream.HandleStreamEvent(QUIC_STREAM_EVENT& streamEvent) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/QuicStream.cs:line 635
   at System.Net.Quic.QuicStream.NativeCallback(QUIC_HANDLE* stream, Void* context, QUIC_STREAM_EVENT* streamEvent) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/QuicStream.cs:line 667
--- End of stack trace from previous location ---
   at System.Net.Quic.ResettableValueTaskSource.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/Internal/ResettableValueTaskSource.cs:line 251
   at System.Net.Quic.QuicStream.ReadAsync(Memory`1 buffer, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/QuicStream.cs:line 308
   at System.Net.Http.Http3RequestStream.ReadResponseContentAsync(HttpResponseMessage response, Memory`1 buffer, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 1208
   --- End of inner exception stack trace ---
   at System.Net.Http.Http3RequestStream.HandleReadResponseContentException(Exception ex, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 1247
   at System.Net.Http.Http3RequestStream.ReadResponseContentAsync(HttpResponseMessage response, Memory`1 buffer, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 1228
   at System.IO.Stream.<CopyToAsync>g__Core|27_0(Stream source, Stream destination, Int32 bufferSize, CancellationToken cancellationToken) in /_/src/libraries/System.Private.CoreLib/src/System/IO/Stream.cs:line 106
   at System.Net.Http.HttpConnectionResponseContent.<SerializeToStreamAsync>g__Impl|6_0(Stream stream, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionResponseContent.cs:line 61
   at System.Net.Http.HttpContent.LoadIntoBufferAsyncCore(Task serializeToStreamTask, MemoryStream tempBuffer) in /_/src/libraries/System.Net.Http/src/System/Net/Http/HttpContent.cs:line 515
   --- End of inner exception stack trace ---
   at System.Net.Http.HttpContent.LoadIntoBufferAsyncCore(Task serializeToStreamTask, MemoryStream tempBuffer) in /_/src/libraries/System.Net.Http/src/System/Net/Http/HttpContent.cs:line 521
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/HttpClient.cs:line 542
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/HttpClient.cs:line 550
   at HttpStress.RequestContext.SendAsync(HttpRequestMessage request, HttpCompletionOption httpCompletion, Nullable`1 token) in C:\app\ClientOperations.cs:line 95
   at HttpStress.ClientOperations.<>c.<<get_Operations>b__1_4>d.MoveNext() in C:\app\ClientOperations.cs:line 288
   at HttpStress.ClientOperations.<>c.<<get_Operations>b__1_4>d.MoveNext() in C:\app\ClientOperations.cs:line 311

2 x HttpIOException with Unknown error code in POST Duplex Slow on Windows

https://dev.azure.com/dnceng-public/public/_build/results?buildId=642813&view=logs&j=0bc77094-9fcd-5c38-f6e4-27d2ae131589&t=4fd93899-d50b-5dc1-f9eb-66ab1b883c22

System.Net.Http.HttpRequestException: Error while copying content to a stream.
 ---> System.Net.Http.HttpIOException: An error occurred while sending the request. (Unknown)
 ---> System.Net.Http.HttpRequestException: An error occurred while sending the request.
 ---> System.Net.Quic.QuicException: Operation aborted.
   at System.Net.Quic.ResettableValueTaskSource.TryComplete(Exception exception, Boolean final) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/Internal/ResettableValueTaskSource.cs:line 177
   at System.Net.Quic.QuicStream.HandleEventShutdownComplete(_SHUTDOWN_COMPLETE_e__Struct& data) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/QuicStream.cs:line 615
   at System.Net.Quic.QuicStream.HandleStreamEvent(QUIC_STREAM_EVENT& streamEvent) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/QuicStream.cs:line 638
   at System.Net.Quic.QuicStream.NativeCallback(QUIC_HANDLE* stream, Void* context, QUIC_STREAM_EVENT* streamEvent) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/QuicStream.cs:line 667
--- End of stack trace from previous location ---
   at System.Net.Quic.ResettableValueTaskSource.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/Internal/ResettableValueTaskSource.cs:line 251
   at System.Net.Quic.QuicStream.WriteAsync(ReadOnlyMemory`1 buffer, Boolean completeWrites, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Quic/src/System/Net/Quic/QuicStream.cs:line 419
   at System.Net.Http.Http3RequestStream.WriteRequestContentAsync(ReadOnlyMemory`1 buffer, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 481
   at HttpStress.ClientOperations.ByteAtATimeNoLengthContent.SerializeToStreamAsync(Stream stream, TransportContext context) in C:\app\ClientOperations.cs:line 615
   at System.Net.Http.HttpContent.<CopyToAsync>g__WaitAsync|56_0(ValueTask copyTask) in /_/src/libraries/System.Net.Http/src/System/Net/Http/HttpContent.cs:line 399
   --- End of inner exception stack trace ---
   --- End of inner exception stack trace ---
   at System.Net.Http.Http3RequestStream.HandleReadResponseContentException(Exception ex, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 1268
   at System.Net.Http.Http3RequestStream.ReadResponseContentAsync(HttpResponseMessage response, Memory`1 buffer, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 1228
   at System.IO.Stream.<CopyToAsync>g__Core|27_0(Stream source, Stream destination, Int32 bufferSize, CancellationToken cancellationToken) in /_/src/libraries/System.Private.CoreLib/src/System/IO/Stream.cs:line 108
   at System.Net.Http.HttpConnectionResponseContent.<SerializeToStreamAsync>g__Impl|6_0(Stream stream, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionResponseContent.cs:line 61
   at System.Net.Http.HttpContent.LoadIntoBufferAsyncCore(Task serializeToStreamTask, MemoryStream tempBuffer) in /_/src/libraries/System.Net.Http/src/System/Net/Http/HttpContent.cs:line 515
   --- End of inner exception stack trace ---
   at System.Net.Http.HttpContent.LoadIntoBufferAsyncCore(Task serializeToStreamTask, MemoryStream tempBuffer) in /_/src/libraries/System.Net.Http/src/System/Net/Http/HttpContent.cs:line 521
   at System.Net.Http.HttpContent.WaitAndReturnAsync[TState,TResult](Task waitTask, TState state, Func`2 returnFunc) in /_/src/libraries/System.Net.Http/src/System/Net/Http/HttpContent.cs:line 833
   at HttpStress.ClientOperations.<>c.<<get_Operations>b__1_8>d.MoveNext() in C:\app\ClientOperations.cs:line 372
--- End of stack trace from previous location ---
   at HttpStress.StressClient.<>c__DisplayClass17_0.<<StartCore>g__RunWorker|0>d.MoveNext() in C:\app\StressClient.cs:line 204

SslStress on `main`

No failures

`release/8.0` branch

HttpStress on `release/8.0`

No failures

SslStress on `release/8.0`

No failures

antonfirsov · 2024-05-09T16:20:21Z

Report 2024-05-09

Looking at runs since 2024-04-26.

`main` (9.0) branch

HttpStress on `main`

There was a temporary issue with dotnet-install that reduced the number of Windows runs, but it is gone now.
Linux builds are failing because of the broken msquic/main build. @CarnaViire will file a fix for that.
Known failures in the remaining runs:

SslStress on `main`

There was a temporary issue with dotnet-install that reduced the number of Windows runs, but it is gone now.
No failures otherwise

`release/8.0` branch

HttpStress on `release/8.0`

No failures

SslStress on `release/8.0`

No failures

antonfirsov · 2024-05-14T16:08:48Z

Report 2024-05-14

Looking at runs since 2024-05-09.

`main` (9.0) branch

HttpStress on `main`

#102085 switched the Linux runs to use msquic v2.3.5 instead of their main branch which fixed #101625.

1 x HttpStress, HTTP/3: content mismatch in POST Duplex Slow #76183
1 x [HttpStress] HttpIOException with HttpRequestError=Unknown and QuicError=OperationAborted on Windows #102202
1 x HttpProtocolException with H3_ID_ERROR in POST ExpectContinue on Linux, which is a failure we didn't see before. Might be the reuslt of the v2.3.5 switch.

https://dev.azure.com/dnceng-public/public/_build/results?buildId=674960&view=logs&j=0bc77094-9fcd-5c38-f6e4-27d2ae131589&t=4fd93899-d50b-5dc1-f9eb-66ab1b883c22

Failure Type 1/2:
System.Net.Http.HttpRequestException: An error occurred while sending the request.
 ---> System.Net.Http.HttpProtocolException: The HTTP/3 server sent invalid data on the connection. HTTP/3 error code 'H3_ID_ERROR' (0x108). (HttpProtocolError)
   at System.Net.Http.Http3RequestStream.ReadFrameEnvelopeAsync(CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 845
   at System.Net.Http.Http3RequestStream.ReadResponseAsync(CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 339
   at System.Net.Http.Http3RequestStream.SendAsync(CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 206
   --- End of inner exception stack trace ---
   at System.Net.Http.Http3RequestStream.SendAsync(CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 301
   at System.Net.Http.Http3RequestStream.SendAsync(CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs:line 322
   at System.Net.Http.Http3Connection.SendAsync(HttpRequestMessage request, Int64 queueStartingTimestamp, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3Connection.cs:line 236
   at System.Net.Http.Http3Connection.SendAsync(HttpRequestMessage request, Int64 queueStartingTimestamp, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3Connection.cs:line 248
   at System.Net.Http.HttpConnectionPool.TrySendUsingHttp3Async(HttpRequestMessage request, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/ConnectionPool/HttpConnectionPool.Http3.cs:line 87
   at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/ConnectionPool/HttpConnectionPool.cs:line 430
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/RedirectHandler.cs:line 30
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/HttpClient.cs:line 530
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/HttpClient.cs:line 550
   at HttpStress.RequestContext.SendAsync(HttpRequestMessage request, HttpCompletionOption httpCompletion, Nullable`1 token) in C:\app\ClientOperations.cs:line 95
   at HttpStress.ClientOperations.<>c.<<get_Operations>b__1_10>d.MoveNext() in C:\app\ClientOperations.cs:line 408
--- End of stack trace from previous location ---
   at HttpStress.StressClient.<>c__DisplayClass17_0.<<StartCore>g__RunWorker|0>d.MoveNext() in C:\app\StressClient.cs:line 204

SslStress on `main`

No failures

`release/8.0` branch

HttpStress on `release/8.0`

No failures

SslStress on `release/8.0`

No failures

antonfirsov · 2024-05-21T16:52:49Z

Report 2024-05-21

Looking at runs since 2024-05-14.

`main` (9.0) branch

HttpStress on `main`

5 x HttpStress, HTTP/3: content mismatch in POST Duplex Slow #76183
7 x [HTTP/3] H3_FRAME_UNEXPECTED in stress tests #87559
1 x [HttpStress] HttpIOException with HttpRequestError=Unknown and QuicError=OperationAborted on Windows #102202
1 x HttpProtocolException with H3_ID_ERROR in POST ExpectContinue on Windows, similar to the failure in the previous report.

SslStress on `main`

1 x transient build failure
No failures otherwise

`release/8.0` branch

HttpStress on `release/8.0`

1 x HttpStress, HTTP/3: content mismatch in POST Duplex Slow #76183

SslStress on `release/8.0`

No failures

antonfirsov · 2024-05-30T14:33:46Z

Report 2024-05-30

Looking at runs since 2024-05-21.

`main` (9.0) branch

HttpStress on `main`

SslStress on `main`

No failures

`release/8.0` branch

HttpStress on `release/8.0`

No failures

SslStress on `release/8.0`

No failures

antonfirsov · 2024-06-06T14:42:42Z

Report 2024-06-06

Looking at runs since 2024-05-30.

`main` (9.0) branch

HttpStress on `main`

[HttpStress] All runs of POST Multipart Data failing #103128 spoiled all runs across all OS-es and versions

SslStress on `main`

1 x SslStress: SocketException (995) WSA_OPERATION_ABORTED on Windows #76044

`release/8.0` branch

HttpStress on `release/8.0`

No failures

SslStress on `release/8.0`

No failures

antonfirsov · 2024-06-11T15:40:17Z

Report 2024-06-11

Looking at runs since 2024-06-06.

`main` (9.0) branch

HttpStress on `main`

4 x HttpStress, HTTP/3: content mismatch in POST Duplex Slow #76183
1 x [HTTP/3] H3_FRAME_UNEXPECTED in stress tests #87559
2 x HttpProtocolException with H3_ID_ERROR in POST ExpectContinue in the same run on Windows, similar to the failure in the previous report, but this time in POST Duplex Slow and GET Parameters

SslStress on `main`

No failures

`release/8.0` branch

HttpStress on `release/8.0`

No failures

SslStress on `release/8.0`

No failures

ManickaP · 2024-06-11T17:41:51Z

2 x HttpProtocolException with H3_ID_ERROR in POST ExpectContinue in the same run on Windows, similar to the failure in the #42211 (comment), but this time in POST Duplex Slow and GET Parameters

Looks like: #98220, Ahmet has a PR up for this in: #103081, but I see it happens in reading the content after SendAsync, so we should address that as well.

antonfirsov · 2024-06-18T15:50:58Z

Report 2024-06-18

Looking at runs since 2024-06-11.

`main` (9.0) branch

HttpStress on `main`

3 x HttpStress, HTTP/3: content mismatch in POST Duplex Slow #76183
1 x [HTTP/3] [HttpStress] QuicException: Stream aborted by peer (258) in (POST Duplex | PUT) Slow #100534

SslStress on `main`

No failures

`release/8.0` branch

HttpStress on `release/8.0`

No failures

SslStress on `release/8.0`

No failures

antonfirsov · 2024-06-25T18:05:13Z

Report 2024-06-25

Looking at runs since 2024-06-18.

`main` (9.0) branch

HttpStress on `main`

2 x HttpStress, HTTP/3: content mismatch in POST Duplex Slow #76183
5 x [HTTP/3] H3_FRAME_UNEXPECTED in stress tests #87559, Windows / Linux
1 x [HTTP/3] [HttpStress] QuicException: Stream aborted by peer (258) in (POST Duplex | PUT) Slow #100534, Windows

SslStress on `main`

No failures

`release/8.0` branch

HttpStress on `release/8.0`

No failures

SslStress on `release/8.0`

No failures

antonfirsov · 2024-07-10T12:28:20Z

Report 2024-07-10

Looking at runs since 2024-06-25.

`main` (9.0) branch

HttpStress on `main`

We assume these issues got fixed with an upstream update starting with 2024-07-09.

SslStress on `main`

No failures

`release/8.0` branch

HttpStress on `release/8.0`

No failures

SslStress on `release/8.0`

No failures

ManickaP added the area-System.Net.Http label Sep 14, 2020

ManickaP added this to the 6.0.0 milestone Sep 14, 2020

ManickaP self-assigned this Sep 14, 2020

Dotnet-GitSync-Bot added the untriaged New issue has not been triaged by the area owner label Sep 14, 2020

ManickaP removed the untriaged New issue has not been triaged by the area owner label Sep 14, 2020

ManickaP mentioned this issue Sep 16, 2020

HTTP stress test improvements #42313

Merged

karelz mentioned this issue Sep 21, 2020

HttpStress tests failing in difference scenarios on sending data to server #40388

Closed

karelz unassigned ManickaP Sep 21, 2020

karelz added the bug label Sep 21, 2020

ManickaP mentioned this issue Mar 22, 2021

Various HTTP Stress test failures on the CI #49999

Closed

antonfirsov mentioned this issue Apr 7, 2021

HttpStress: connection failures with HTTP 1.1 caused by Windows Firewall #50854

Closed

antonfirsov mentioned this issue May 6, 2021

HttpStress: Disable firewall in Windows runs #52381

Merged

antonfirsov mentioned this issue May 12, 2021

HttpStress: also disable firewall for HTTP2 runs #52643

Merged

antonfirsov added a commit that referenced this issue May 18, 2021

HttpStress: also disable firewall for HTTP2 (#52643)

8fb2eb5

We decided that we need to rule out the Windows Firewall as a cause for further failure types documented in #42211.

karelz modified the milestones: 6.0.0, 7.0.0 Jul 29, 2021

This comment has been minimized.

Sign in to view

ManickaP mentioned this issue Oct 20, 2023

HttpStress, HTTP/3: content mismatch in POST Duplex Slow #76183

Open

karelz mentioned this issue Oct 31, 2023

[SslStress] SocketException (111): Connection refused #94222

Closed

antonfirsov mentioned this issue May 30, 2024

[HttpStress] H3_FRAME_ERROR in Post Duplex Slow #102880

Closed

antonfirsov mentioned this issue Jun 11, 2024

SslStress: SocketException (995) WSA_OPERATION_ABORTED on Windows #76044

Closed

ManickaP mentioned this issue Jun 11, 2024

[HTTP/3] map QuicException to OCE in case of a cancellation #98220

Closed

antonfirsov modified the milestones: 9.0.0, 10.0.0 Jun 25, 2024

Http Stress Status Report #42211

Http Stress Status Report #42211

Comments

ManickaP commented Sep 14, 2020 • edited Loading

Http Stress Status Report

What we've run so far:

HTTP 2.0 Error Statistics

HTTP 1.1 Error Statistics

What we need to run:

Existing issues, root caused:

Discovered exceptions, not-investigated:

Distributable tasks by priority:

Tips and Tricks for investigations:

ghost commented Sep 14, 2020

antonfirsov commented Mar 22, 2021

antonfirsov commented May 10, 2021 • edited Loading

antonfirsov commented Jun 28, 2021

karelz commented Jul 29, 2021

antonfirsov commented Jul 29, 2021 • edited by karelz Loading

Report 7/29

antonfirsov commented Aug 5, 2021

Report 8/5

antonfirsov commented Aug 13, 2021

Report 8/13

antonfirsov commented Aug 24, 2021 • edited Loading

Report 8/24

ManickaP commented Aug 24, 2021

antonfirsov commented Aug 24, 2021 • edited Loading

This comment has been minimized.

karelz commented Aug 25, 2021

antonfirsov commented Oct 12, 2023

Report 2023-10-12

main (9.0) branch

release/8.0 branch

karelz commented Oct 19, 2023 • edited Loading

Report 2023-10-19

main (9.0) branch

release/8.0 branch

antonfirsov commented Oct 31, 2023 • edited Loading

Report 2023-10-31

main (9.0) branch

release/8.0 branch

antonfirsov commented Nov 9, 2023

Report 2023-11-09

main (9.0) branch

release/8.0 branch

antonfirsov commented Jan 4, 2024

Report 2024-01-04

main (9.0) branch

release/8.0 branch

antonfirsov commented Jan 18, 2024

Report 2024-01-18

main (9.0) branch

release/8.0 branch

antonfirsov commented Feb 15, 2024

Report 2024-02-15

main (9.0) branch

release/8.0 branch

antonfirsov commented Feb 29, 2024 • edited Loading

Report 2024-02-29

main (9.0) branch

release/8.0 branch

antonfirsov commented Mar 19, 2024

Report 2024-03-19

main (9.0) branch

release/8.0 branch

antonfirsov commented Mar 26, 2024

Report 2024-03-26

main (9.0) branch

release/8.0 branch

antonfirsov commented Apr 2, 2024

Report 2024-04-02

main (9.0) branch

release/8.0 branch

antonfirsov commented Apr 9, 2024

Report 2024-04-09

main (9.0) branch

release/8.0 branch

antonfirsov commented Apr 26, 2024 • edited Loading

Report 2024-04-26

ManickaP commented Sep 14, 2020 •

edited

Loading

antonfirsov commented May 10, 2021 •

edited

Loading

antonfirsov commented Jul 29, 2021 •

edited by karelz

Loading

antonfirsov commented Aug 24, 2021 •

edited

Loading

antonfirsov commented Aug 24, 2021 •

edited

Loading

`main` (9.0) branch

`release/8.0` branch

karelz commented Oct 19, 2023 •

edited

Loading

`main` (9.0) branch

`release/8.0` branch

antonfirsov commented Oct 31, 2023 •

edited

Loading

`main` (9.0) branch

`release/8.0` branch

`main` (9.0) branch

`release/8.0` branch

`main` (9.0) branch

`release/8.0` branch

`main` (9.0) branch

`release/8.0` branch

`main` (9.0) branch

`release/8.0` branch

antonfirsov commented Feb 29, 2024 •

edited

Loading

`main` (9.0) branch

`release/8.0` branch

`main` (9.0) branch

`release/8.0` branch

`main` (9.0) branch

`release/8.0` branch

`main` (9.0) branch

`release/8.0` branch

`main` (9.0) branch

`release/8.0` branch

antonfirsov commented Apr 26, 2024 •

edited

Loading

`main` (9.0) branch

`release/8.0` branch

antonfirsov commented May 9, 2024 •

edited

Loading

`main` (9.0) branch

`release/8.0` branch

`main` (9.0) branch