Flaky Test: TestKeepAliveTimeout

[davidfowl]

Latest failure message

[TestKeepAliveTimeout] System.IO.IOException : Unable to transfer data on the transport connection: An established connection was aborted by the software in your host machine.
---- System.Net.Sockets.SocketException : An established connection was aborted by the software in your host machine
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.IO.StreamWriter.<FlushAsyncInternal>d__65.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.AspNetCore.Testing.TestConnection.<Send>d__11.MoveNext() in C:\BuildAgent\work\33bdfc1cae7b2a38\.r\KestrelHttpServer\test\shared\TestConnection.cs:line 68
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.AspNetCore.Server.Kestrel.FunctionalTests.KeepAliveTimeoutTests.<ConnectionKeptAliveBetweenRequests>d__5.MoveNext() in C:\BuildAgent\work\33bdfc1cae7b2a38\.r\KestrelHttpServer\test\Microsoft.AspNetCore.Server.Kestrel.FunctionalTests\KeepAliveTimeoutTests.cs:line 73
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.AspNetCore.Server.Kestrel.FunctionalTests.KeepAliveTimeoutTests.<<TestKeepAliveTimeout>b__3_0>d.MoveNext() in C:\BuildAgent\work\33bdfc1cae7b2a38\.r\KestrelHttpServer\test\Microsoft.AspNetCore.Server.Kestrel.FunctionalTests\KeepAliveTimeoutTests.cs:line 48
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
----- Inner Stack Trace -----

5/11/17: http://aspnetci/viewLog.html?buildId=218866&buildTypeId=XPlat_Windows_Win7_Universe&tab=buildLog&filter=err#_focus=288981

Original failure message

I ran the functional tests in a loop and it failed with the following:

[xUnit.net 00:01:03.2382155]     TestKeepAliveTimeout [FAIL]
[xUnit.net 00:01:03.2398021]       System.IO.IOException : Unable to transfer data on the transport connection: Broken pipe.
[xUnit.net 00:01:03.2399534]       ---- System.Net.Sockets.SocketException : Broken pipe
[xUnit.net 00:01:03.2413298]       Stack Trace:
[xUnit.net 00:01:03.2428599]            at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
[xUnit.net 00:01:03.2430077]            at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
[xUnit.net 00:01:03.2430613]            at System.IO.StreamWriter.<FlushAsyncInternal>d__65.MoveNext()
[xUnit.net 00:01:03.2431238]         --- End of stack trace from previous location where exception was thrown ---
[xUnit.net 00:01:03.2432002]            at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
[xUnit.net 00:01:03.2433100]            at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
[xUnit.net 00:01:03.2433561]            at Microsoft.AspNetCore.Testing.TestConnection.<Send>d__11.MoveNext()
[xUnit.net 00:01:03.2433751]         --- End of stack trace from previous location where exception was thrown ---
[xUnit.net 00:01:03.2433994]            at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
[xUnit.net 00:01:03.2434906]            at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
[xUnit.net 00:01:03.2435713]            at Microsoft.AspNetCore.Server.Kestrel.FunctionalTests.KeepAliveTimeoutTests.<ConnectionKeptAliveBetweenRequests>d__5.MoveNext()
[xUnit.net 00:01:03.2436558]         --- End of stack trace from previous location where exception was thrown ---
[xUnit.net 00:01:03.2437408]            at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
[xUnit.net 00:01:03.2438098]            at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
[xUnit.net 00:01:03.2438414]            at Microsoft.AspNetCore.Server.Kestrel.FunctionalTests.KeepAliveTimeoutTests.<TestKeepAliveTimeout>d__3.MoveNext()
[xUnit.net 00:01:03.2439671]         --- End of stack trace from previous location where exception was thrown ---
[xUnit.net 00:01:03.2440045]            at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
[xUnit.net 00:01:03.2441334]            at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
[xUnit.net 00:01:03.2441633]         --- End of stack trace from previous location where exception was thrown ---
[xUnit.net 00:01:03.2442079]            at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
[xUnit.net 00:01:03.2442491]            at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
[xUnit.net 00:01:03.2443089]         --- End of stack trace from previous location where exception was thrown ---
[xUnit.net 00:01:03.2443845]            at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
[xUnit.net 00:01:03.2444837]            at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
[xUnit.net 00:01:03.2445164]         ----- Inner Stack Trace -----
[xUnit.net 00:01:03.2445361]
Failed   TestKeepAliveTimeout
Error Message:
 System.IO.IOException : Unable to transfer data on the transport connection: Broken pipe.
---- System.Net.Sockets.SocketException : Broken pipe
Stack Trace:
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.IO.StreamWriter.<FlushAsyncInternal>d__65.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.AspNetCore.Testing.TestConnection.<Send>d__11.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.AspNetCore.Server.Kestrel.FunctionalTests.KeepAliveTimeoutTests.<ConnectionKeptAliveBetweenRequests>d__5.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.AspNetCore.Server.Kestrel.FunctionalTests.KeepAliveTimeoutTests.<TestKeepAliveTimeout>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

[davidfowl]

Happend again on travis in dev

[davidfowl]

These tests used to fail because of slowness #1157.

/cc @CesarBS

[Tratcher]

Also failing on Win7 and Ubuntu CIs

[muratg]

Assigning to @halter73 for investigation.

[Tratcher]

@halter73 this is failing consistently. Please disable it if it's not resolved by end of day.

[Tratcher]

http://aspnetci/viewLog.html?buildId=204056&tab=buildResultsDiv&buildTypeId=XPlat_Windows_Win7_Universe

System.IO.IOException : Unable to transfer data on the transport connection: An established connection was aborted by the software in your host machine.
---- System.Net.Sockets.SocketException : An established connection was aborted by the software in your host machine
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.IO.StreamWriter.<FlushAsyncInternal>d__65.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.AspNetCore.Testing.TestConnection.<Send>d__11.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.AspNetCore.Server.Kestrel.FunctionalTests.KeepAliveTimeoutTests.<ConnectionKeptAliveBetweenRequests>d__5.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.AspNetCore.Server.Kestrel.FunctionalTests.KeepAliveTimeoutTests.<TestKeepAliveTimeout>d__3.MoveNext()

[halter73]

I think this test is far more stable post a053ca4 and e4ba1d0. We can reopen if we see more failures.

[mikeharder]

This is still failing consistently on Win7 for preview1:

http://aspnetci/viewLog.html?buildId=217097&buildTypeId=XPlat_Windows_Win7_Universe&tab=buildLog

[muratg]

We'll disable the test for preview1. And actually try to fix it in preview2.

[mikeharder]

Test disabled for Preview1: #1817

[muratg]

@CesarBS could you reenable this and see if it still fails?

[cesarblum]

Looks like it's been failing again, on Windows 7 only.

[cesarblum]

As I suspected, here's the root of the flakiness:

| Microsoft.AspNetCore.Server.Kestrel.FunctionalTests.KeepAliveTimeoutTests Debug: Short delay lasted 11.072 seconds instead of 3 seconds.

The server is closing the connection because of that, causing the test to fail.

[cesarblum]

I'll watch the CI for a couple days before closing this.