Microsoft.AspNetCore.Mvc.Testing : The request was aborted or the pipeline has finished on TeamCity

[mii9000]

Describe the bug

Our Integration Suite on TeamCity keeps on failing randomly but runs fine locally from Visual Studio and command line as well.

To Reproduce

Steps to reproduce the behavior:

1. Using this version of ASP.NET Core '2.1.1'
2. Run tests on TeamCity

Expected behavior

Assert errors but no errors from TestHost

Screenshots

Stack trace from TeamCity:

 Failed   Agoda.Ycs.Gateway.IntegrationTests.GetCakesEndpointTest.CakesEndpointTest(url: "/api/Cakes/1")
 Error Message:
  System.IO.IOException : The request was aborted or the pipeline has finished
 Stack Trace:
    at Microsoft.AspNetCore.TestHost.ResponseStream.CheckNotComplete()
    at Microsoft.AspNetCore.TestHost.ResponseStream.WriteAsync(Byte[] buffer, Int32 offset, Int32 count, CancellationToken cancellationToken)
    at Microsoft.AspNetCore.WebUtilities.HttpResponseStreamWriter.FlushInternalAsync(Boolean flushEncoder)
    at Microsoft.AspNetCore.Mvc.Formatters.JsonOutputFormatter.WriteResponseBodyAsync(OutputFormatterWriteContext context, Encoding selectedEncoding)
    at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeResultAsync(IActionResult result)
    at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeNextResultFilterAsync[TFilter,TFilterAsync]()
    at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Rethrow(ResultExecutedContext context)
    at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.ResultNext[TFilter,TFilterAsync](State& next, Scope& scope, Object& state, Boolean& isCompleted)
    at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeResultFilters()
    at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeNextResourceFilter()
    at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Rethrow(ResourceExecutedContext context)
    at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
    at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeFilterPipelineAsync()
    at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeAsync()
    at Microsoft.AspNetCore.Builder.RouterMiddleware.Invoke(HttpContext httpContext)
    at Agoda.Ycs.Gateway.Api.Middlewares.MeasurementsLoggingMiddleware.Invoke(HttpContext context, IOptionsSnapshot`1 adpMessagingClientSettings)
    at Microsoft.AspNetCore.StaticFiles.StaticFileMiddleware.Invoke(HttpContext context)
    at Swashbuckle.AspNetCore.SwaggerUI.SwaggerUIIndexMiddleware.Invoke(HttpContext httpContext)
    at Swashbuckle.AspNetCore.Swagger.SwaggerMiddleware.Invoke(HttpContext httpContext)
    at Microsoft.AspNetCore.StaticFiles.StaticFileMiddleware.Invoke(HttpContext context)
    at Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddleware.Invoke(HttpContext context)
    at Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddleware.Invoke(HttpContext context)
    at Microsoft.AspNetCore.TestHost.HttpContextBuilder.<>c__DisplayClass10_0.<<SendAsync>b__0>d.MoveNext()
 --- End of stack trace from previous location where exception was thrown ---
    at Microsoft.AspNetCore.TestHost.ClientHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
    at Microsoft.AspNetCore.Mvc.Testing.Handlers.CookieContainerHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
    at Microsoft.AspNetCore.Mvc.Testing.Handlers.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
    at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
    at Agoda.Ycs.Gateway.IntegrationTests.GetEndpointTestBase`1.EndpointsReturnSuccessAndCorrectContentType(String url) in /opt/buildagent/work/69f9ec429751afcd/tests/Agoda.Ycs.Gateway.IntegrationTests/GetEndpointTestBase.cs:line 16
    at Agoda.Ycs.Gateway.IntegrationTests.GetCakesEndpointTest.CakesEndpointTest(String url) in /opt/buildagent/work/69f9ec429751afcd/tests/Agoda.Ycs.Gateway.IntegrationTests/GetCakesEndpointTest.cs:line 15

Additional context

• Teamcity agents are running Linux, version 3.10.0-862.3.3.el7.x86_64

[pranavkm]

@Tratcher does this ring a bell?

[Tratcher]

You might see this if the client timed out or aborted the request.

Can you share CakesEndpointTest and related code for reference?

[mii9000]

@Tratcher Thank you for your reply. Please find the full code below.
I understand its a timeout issue and I can see in TeamCity logs that randomly some requests would take more than 2 mins or so. But I cannot understand why I cannot repro it locally though.

namespace IntegrationTests
{
    public abstract class BaseIntegrationTest<TStartup> where TStartup: class
    {
        protected static WebApplicationFactory<TStartup> _factory;

        protected BaseIntegrationTest()
        {
            if (_factory is null)
                _factory = new WebApplicationFactory<TStartup>();
        }

        protected async Task AssertSuccessWithBody(HttpResponseMessage response)
        {
            await DoAssertSuccessWithValidJson(response);
        }
    }    

    public abstract class GetEndpointTestBase<TStartup> : BaseIntegrationTest<TStartup>, 
        IClassFixture<WebApplicationFactory<TStartup>> where TStartup : class
    {
        protected async Task EndpointsReturnSuccessAndCorrectContentType(string url)
        {
            // Arrange
            var client = _factory.CreateClient();

            // Act
            var response = await client.GetAsync(url);

            // Assert
            await AssertSuccessWithBody(response);
        }
    }

    public class GetCakesEndpointTest : GetEndpointTestBase<Startup>
    {
        [Theory]
        [InlineData("/api/Cakes/1")]
        public async Task CakesEndpointTest(string url)
        {
            await EndpointsReturnSuccessAndCorrectContentType(url);
        }
    }
}

[Tratcher]

Sounds like a deadlock in your app. XUnit is known for limiting the number of threads allowed for a test, so if you accidentally block any then your test can starve. If you can get a dump or attach a debugger then check for blocked threads.

[mii9000]

@Tratcher How can I get a dump while the tests are being run on TeamCity? Any ideas would be appreciated.

[Tratcher]

@pranavkm ?

[pranavkm]

How can I get a dump while the tests are being run on TeamCity? Any ideas would be appreciated.

I guess log on the TeamCity agent and produce a dump from the process?

[mii9000]

Some of the tests were connecting to a dependent service which was timing out for some configuration reasons. Could not determine how to produce logs and diagnose the issue but not happening anymore. Thanks everyone.

[csMACnz]

I found this issue from similar symptoms and wanted to add an extra bit of information that helped me diagnose, for the next person who sees this.

Something in the request was taking too long, which was triggering the HttpClient's default timeout of 100 seconds. AspNetCore Cancellation Tokens don't cause an immediate failure, but instead will trigger when you try to write your response, giving the error above "The request was aborted or the pipeline has finished" at that point.

Adding internal timeouts to the internal Http calls inside your service helps to find the cause of the slow requests, and trigger the failure where it is slow rather than after it is slow.

Otherwise we basically had the same issue as @ibrahim-islam (slow upstream responses)

[mii9000]

@csMACnz Can you please share some code samples as to how you have changed your tests to workaround this error?

[artisticcheese]

Increasing default HTTPClient timeout for me fixed it.
client.Timeout = System.TimeSpan.FromMinutes(3);