System.InvalidOperationException: Reading is already in progress thrown by api-gateway like application

[lalernehl]

This is how my C# app works. I'm getting a request and I want to pass it to a downstream service, so yes, I'm acting as a api-gateway. Problem is that I'm seeing System.InvalidOperationException: Reading is already in progress exceptions.

The code goes like this:

// map request
HttpRequest request = GetOriginalRequest();
var requestMessage = new HttpRequestMessage
{
    Content = MapContent(request),
    Method = MapMethod(request),
    RequestUri = MapUri(request)
};
...
// send request
var response = await httpClient.SendAsync(requestMessage); // throws here!!!

The MapContent method looks like this:

...
request.EnableBuffering(); // it throws if I don't do this
request.Body.Seek(0, SeekOrigin.Begin);
var content = new StreamContent(request.Body);
content.Headers.ContentLength = request.ContentLength;
return new StreamContent(request.Body);

This is the Exception:

System.InvalidOperationException: Reading is already in progress.
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.Http1ContentLengthMessageBody.VerifyIsNotReading()
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.Http1ContentLengthMessageBody.TryReadInternal(ReadResult& readResult)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.Http1MessageBody.OnConsumeAsync()

I have obviously omitted many details and code but this is the essence.

Any idea how to reuse the stream to pass it to another service? THE CONTENT CANNOT BE COPIED since this will have a big performance degradation, so this is not acceptable var content = new ByteArrayContent(await ToByteArray(request.Body)); . It will be awesome also not to use EnableBuffering since it uses files, but I'm not sure if I can, or the impact of it (that's one of the things I wanted to test with this new code).

Thanks a lot in advanced!

Further technical details

• ASP.NET Core version :
  .NET 5.0
• Include the output of dotnet --info

.NET SDK (reflecting any global.json):
 Version:   5.0.203
 Commit:    383637d63f

Runtime Environment:
 OS Name:     Windows
 OS Version:  10.0.19042
 OS Platform: Windows
 RID:         win10-x64
 Base Path:   C:\Program Files\dotnet\sdk\5.0.203\

Host (useful for support):
  Version: 5.0.6
  Commit:  478b2f8c0e

.NET SDKs installed:
  3.1.409 [C:\Program Files\dotnet\sdk]
  5.0.104 [C:\Program Files\dotnet\sdk]
  5.0.201 [C:\Program Files\dotnet\sdk]
  5.0.203 [C:\Program Files\dotnet\sdk]

.NET runtimes installed:
  Microsoft.AspNetCore.All 2.1.26 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.1.28 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.App 2.1.26 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.1.28 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 3.1.13 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 3.1.15 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 5.0.4 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 5.0.6 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 2.1.26 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.28 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 3.1.0 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 3.1.13 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 3.1.15 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 5.0.4 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 5.0.6 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.WindowsDesktop.App 3.1.13 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 3.1.15 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 5.0.4 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 5.0.6 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]

• The IDE (VS / VS Code/ VS4Mac) you're running on, and its version
  Microsoft Visual Studio Professional 2019. Version 16.9.1

[davidfowl]

@halter73 @Tratcher

[Tratcher]

What's happening here? The ContentLength should be set on the StreamContent you're returning. What's the other content object?

content.Headers.ContentLength = request.ContentLength;
return new StreamContent(request.Body);

We've had a few reports of EnableBuffering causing conflicts with StreamContent, but not this specific error. Does it work if you remove EnableBuffering?

It's not clear from your description if buffering is required for your scenario. Do you need to send the content multiple places?

[BrennanConroy]

Related issue #17840 we believe

[lalernehl]

No, I don't need to send the content multiple times, but something (outside of my knowledge) seems to be reading the content, reason why I had to enable buffering, BUT as I stated before, I'd very much like to NOT have that.

I tried adding the content-length but then I started getting The stream was already consumed. It cannot be read again and I blamed that line for that.

I think it's worth saying that under certain scenarios, like local testing, I saw different outcomes. The issue is reproduceable under semi-heavy traffic (+100rps) and the downstream service is a go service.

[BrennanConroy]

Can you provide the full stack traces of the exceptions you're observing as well as when you're seeing them.

[Tratcher]

See dotnet/runtime#53914 for a similar scenario and suggestions.

[halter73]

System.InvalidOperationException: Reading is already in progress.
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.Http1ContentLengthMessageBody.VerifyIsNotReading()
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.Http1ContentLengthMessageBody.TryReadInternal(ReadResult& readResult)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.Http1MessageBody.OnConsumeAsync()

So we normally see error when the app isn't properly awaiting a request body read. In this case, it's HttpClient that's reading the request body and the SendAsync is apparently failing.

I wonder if HttpClient didn't wait for its read from Kestrel's request body to complete and if this is the same issue as dotnet/runtime#53914

[halter73]

Related issue #17840 we believe

@BrennanConroy I didn't mean for that to be this issue. I meant for this to be on #33373, but looking at that more closely now, I think HttpRequestPipeReader needs to be a little more thread safe.

[lalernehl]

What's happening here? The ContentLength should be set on the StreamContent you're returning. What's the other content object?

content.Headers.ContentLength = request.ContentLength;
return new StreamContent(request.Body);

We've had a few reports of EnableBuffering causing conflicts with StreamContent, but not this specific error. Does it work if you remove EnableBuffering?

It's not clear from your description if buffering is required for your scenario. Do you need to send the content multiple places?

Sorry @Tratcher I had a typo in my code and I have updated already. It was supposed to look like this:

request.EnableBuffering(); // it throws if I don't do this
request.Body.Seek(0, SeekOrigin.Begin);
var content = new StreamContent(request.Body);   // <----- NEW LINE
content.Headers.ContentLength = request.ContentLength;
return new StreamContent(request.Body);

[lalernehl]

Can you provide the full stack traces of the exceptions you're observing as well as when you're seeing them.

@BrennanConroy here are some logs I got using the code as follow:

//request.EnableBuffering();    <--- COMMENTED OUT
//request.Body.Seek(0, SeekOrigin.Begin);    <--- COMMENTED OUT
var content = new StreamContent(request.Body);
content.Headers.ContentLength = request.ContentLength;

Stacktrace:

System.Net.Http.HttpRequestException: An error occurred while sending the request.
 ---> System.InvalidOperationException: The stream was already consumed. It cannot be read again.
   at System.Net.Http.StreamContent.PrepareContent()
   at System.Net.Http.StreamContent.SerializeToStreamAsync(Stream stream, TransportContext context, CancellationToken cancellationToken)
   at System.Net.Http.HttpContent.InternalCopyToAsync(Stream stream, TransportContext context, CancellationToken cancellationToken)
   at System.Net.Http.HttpContent.CopyToAsync(Stream stream, TransportContext context, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnection.SendRequestContentAsync(HttpRequestMessage request, HttpContentWriteStream stream, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnection.SendRequestContentWithExpect100ContinueAsync(HttpRequestMessage request, Task`1 allowExpect100ToContinueTask, HttpContentWriteStream stream, Timer expect100Timer, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.DiagnosticsHandler.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at Microsoft.Extensions.Http.Logging.LoggingHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at Polly.Timeout.AsyncTimeoutEngine.ImplementationAsync[TResult](Func`3 action, Context context, CancellationToken cancellationToken, Func`2 timeoutProvider, TimeoutStrategy timeoutStrategy, Func`5 onTimeoutAsync, Boolean continueOnCapturedContext)
   at Polly.Timeout.AsyncTimeoutEngine.ImplementationAsync[TResult](Func`3 action, Context context, CancellationToken cancellationToken, Func`2 timeoutProvider, TimeoutStrategy timeoutStrategy, Func`5 onTimeoutAsync, Boolean continueOnCapturedContext)
   at Polly.AsyncPolicy`1.ExecuteAsync(Func`3 action, Context context, CancellationToken cancellationToken, Boolean continueOnCapturedContext)
   at Microsoft.Extensions.Http.PolicyHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at Polly.Retry.AsyncRetryEngine.ImplementationAsync[TResult](Func`3 action, Context context, CancellationToken cancellationToken, ExceptionPredicates shouldRetryExceptionPredicates, ResultPredicates`1 shouldRetryResultPredicates, Func`5 onRetryAsync, Int32 permittedRetryCount, IEnumerable`1 sleepDurationsEnumerable, Func`4 sleepDurationProvider, Boolean continueOnCapturedContext)
   at Polly.AsyncPolicy`1.ExecuteAsync(Func`3 action, Context context, CancellationToken cancellationToken, Boolean continueOnCapturedContext)
   at Microsoft.Extensions.Http.PolicyHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at Polly.CircuitBreaker.AsyncCircuitBreakerEngine.ImplementationAsync[TResult](Func`3 action, Context context, CancellationToken cancellationToken, Boolean continueOnCapturedContext, ExceptionPredicates shouldHandleExceptionPredicates, ResultPredicates`1 shouldHandleResultPredicates, ICircuitController`1 breakerController)
   at Polly.AsyncPolicy`1.ExecuteAsync(Func`3 action, Context context, CancellationToken cancellationToken, Boolean continueOnCapturedContext)
   at Microsoft.Extensions.Http.PolicyHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at Microsoft.Extensions.Http.Logging.LoggingScopeHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.SendAsyncCore(HttpRequestMessage request, HttpCompletionOption completionOption, Boolean async, Boolean emitTelemetryStartStop, CancellationToken cancellationToken)
   at GatewayHttpRequester.GetResponse(BackendContext context) in /code_path/GatewayHttpRequester.cs:line 00

[Tratcher]

InvalidOperationException: The stream was already consumed. It cannot be read again.

Ok, I'd misread some of your earlier description, this makes sense now. There are situations where HttpClient does need to retry sending the request body. If the body wasn't buffered then you get the above error on the retry. Some of the retry conditions are discussed in dotnet/runtime#53914 (comment). 100-continue can be used to mitigate some of these retry situations, but they can't be completely avoided.

Edit That doesn't directly explain the other error System.InvalidOperationException: Reading is already in progress.