PipeWriter.GetMemory() InvalidOperationException: Writing is not allowed after writer was completed.

Describe the bug

Under extreme load, we have noticed that the PipeWriter.GetMemory() method will fail with an InvalidOperationException of "Writing is not allowed after writer was completed."

To Reproduce

We are processing ~4,000-6,000 requests/second over a TCP connection, for which I have written a custom ConnectionHandler. The request sizes vary from ~150 bytes to ~100k, but can be replicated with the smaller request size. For reference, the response is ~3-5 times that of the request. This issue only started to show up after we were able to run performance tests near the server's maximum capacity.

At first I thought that it was just a closed connection, but these exceptions are occurring while the CancellationToken of ConnectionContext.ConnectionClosed has an IsCancellationRequested property value of false. It is possible that the connection is closed, but this property is not showing it as such. Due to the way that the loss of a connection works, it creates a race condition and we have not had any success at reproducing the issue in a debug session.

Exceptions (if any)

InvalidOperationException: Writing is not allowed after writer was completed.

Further technical details

• ASP.NET Core version: 3.1.301
• This is running on AWS Linux, latest release, netcoreapp3.1, Self-Contained, linux-x64

[davidfowl]

We need to see the code. Can you share a repro with us?

It will take me a few days to get you an application to replicate this, but I can share the method where it is occuring:

private async Task SendResponseAsync(
	PipeWriter writer,
	HttpResult httpResult,
	bool isHttpRequest,
	CancellationToken closedToken
)
{
	if (closedToken.IsCancellationRequested)
	{
		return;
	}

	int totalBytes;
	Memory<byte> outputMemory;

	if (isHttpRequest)
	{
		string httpResponseHeader = string.Format(
			HTTP_RESPONSE_FORMAT,
			httpResult.HttpStatusCode.ToString("D"),
			HttpStatusDescription.Get(httpResult.HttpStatusCode),
			httpResult.ContentType,
			httpResult.Response.Length
		);

		byte[] httpResponseHeaderBytes = Encoding.UTF8.GetBytes(httpResponseHeader);
		totalBytes = httpResult.Response.Length + httpResponseHeaderBytes.Length;

		if (closedToken.IsCancellationRequested)
		{
			return;
		}

		outputMemory = writer.GetMemory(totalBytes);
		httpResponseHeaderBytes.CopyTo(outputMemory);
		httpResult.Response.CopyTo(outputMemory.Slice(httpResponseHeaderBytes.Length));
	}
	else
	{
		totalBytes = httpResult.Response.Length;

		if (closedToken.IsCancellationRequested)
		{
			return;
		}

		outputMemory = writer.GetMemory(totalBytes);
		httpResult.Response.CopyTo(outputMemory);
	}

	if (closedToken.IsCancellationRequested)
	{
		return;
	}

	writer.Advance(totalBytes);
	await writer.FlushAsync();

#if DEBUG
	_logger.LogDebug(
		"outputMemory {0} bytes ({1} bytes requested)",
		outputMemory.Span.Length,
		totalBytes
	);
#endif
}

The exception being thrown occurs every 5-10 seconds and is:

System.InvalidOperationException: Writing is not allowed after writer was completed.
   at System.IO.Pipelines.ThrowHelper.ThrowInvalidOperationException_NoWritingAllowed()
   at System.IO.Pipelines.Pipe.GetMemory(Int32 sizeHint)
   at [snip]ApiTcpHandler.<>c__DisplayClass32_2.<<OnConnectedAsync>g__sendResponseAsync|3>d.MoveNext() in /[snip]ApiTcpHandler.cs:line 477
--- End of stack trace from previous location where exception was thrown ---
   at [snip]ApiProcessor.ProcessRequestAsync[TMethod](String serviceType, IPAddress remoteIpAddress, CreateBcapRequestAsync`1 createBcapRequestAsync, SendResponseAsync sendResponseAsync) in /[snip]ApiProcessor.cs:line 382

I am using the following to filter out the exceptions as defined by this issue. When using a DEBUG build, it always logs "false":

catch (InvalidOperationException ex2) when ("Writing is not allowed after writer was completed.".Equals(ex2.Message, StringComparison.InvariantCultureIgnoreCase))
{
#if DEBUG
	_logger.LogWarning("{0} (closed: {1})", ex.Message, closedToken.IsCancellationRequested);
#endif
}

The GetMemory and Advance methods are periodically throwing exceptions that should not be possible. Here are some new exceptions that we discovered today:

System.ArgumentOutOfRangeException: Specified argument was out of the range of valid values. (Parameter 'bytes')
   at System.IO.Pipelines.ThrowHelper.ThrowArgumentOutOfRangeException(ExceptionArgument argument)
   at System.IO.Pipelines.Pipe.Advance(Int32 bytes)
   at [snip].SendResponseAsync(PipeWriter writer, LogResult logResult, Boolean isHttpRequest, CancellationToken closedToken) in /[snip].cs:line 689
   at [snip]ApiTcpHandler.<>c__DisplayClass32_2.<<OnConnectedAsync>g__sendResponseAsync|3>d.MoveNext() in /[snip].cs:line 422
--- End of stack trace from previous location where exception was thrown ---
   at [snip]ApiProcessor.ProcessRequestAsync[TMethod](String serviceType, IPAddress remoteIpAddress, CreateBcapRequestAsync`1 createBcapRequestAsync, SendResponseAsync sendResponseAsync) in /[snip].cs:line 382

System.ArgumentException: Destination is too short. (Parameter 'destination')
   at [snip]ApiTcpHandler.SendResponseAsync(PipeWriter writer, LogResult logResult, Boolean isHttpRequest, CancellationToken closedToken) in /[snip].cs:line 689
   at [snip]ApiTcpHandler.<>c__DisplayClass32_2.<<OnConnectedAsync>g__sendResponseAsync|3>d.MoveNext() in /[snip].cs:line 422
--- End of stack trace from previous location where exception was thrown ---
   at [snip]ApiProcessor.ProcessRequestAsync[TMethod](String serviceType, IPAddress remoteIpAddress, CreateBcapRequestAsync`1 createBcapRequestAsync, SendResponseAsync sendResponseAsync) in /[snip].cs:line 382	

The last one is also demonstrating a StackTrace failure, as the line that is throwing that exception is one of the following lines:

• httpResponseHeaderBytes.CopyTo(outputMemory);
• httpResult.Response.CopyTo(outputMemory.Slice(httpResponseHeaderBytes.Length));
• httpResult.Response.CopyTo(outputMemory);

All of these exceptions are random, with these last 2 occurring approximate once every few minutes. If you still need a project to replicate the issues, I can continue to work on one as time permits.

[BrennanConroy]

We can't figure out how the exceptions are occurring from this method alone unfortunately. Looking forward to a repro app.

This issue has been automatically marked as stale because it has been marked as requiring author feedback but has not had any activity for 4 days. It will be closed if no further activity occurs within 3 days of this comment. If it is closed, feel free to comment when you are able to provide the additional information and we will re-investigate.

See our Issue Management Policies for more information.