Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Announcement] AllowSynchronousIO disabled in all servers #7644

Open
Tratcher opened this issue Feb 16, 2019 · 54 comments
Open

[Announcement] AllowSynchronousIO disabled in all servers #7644

Tratcher opened this issue Feb 16, 2019 · 54 comments

Comments

@Tratcher
Copy link
Contributor

@Tratcher Tratcher commented Feb 16, 2019

AllowSynchronousIO is a option in each server that enables or disables sync IO APIs like HttpReqeuest.Body.Read, HttpResponse.Body.Write, Stream.Flush, etc.. These APIs have long been a source of thread starvation and application hangs. Starting in 3.0.0-preview3 these are disabled by default.

Affected servers:

  • Kestrel
  • HttpSys
  • IIS in-process
  • TestServer

Expect errors similar to:

  • Synchronous operations are disallowed. Call ReadAsync or set AllowSynchronousIO to true instead.
  • Synchronous operations are disallowed. Call WriteAsync or set AllowSynchronousIO to true instead.
  • Synchronous operations are disallowed. Call FlushAsync or set AllowSynchronousIO to true instead.

Each server has a AllowSynchronousIO option that controls this behavior and the default for all of them is now false.

The behavior can also be overridden on a per request basis as a temporary mitigation.

var syncIOFeature = HttpContext.Features.Get<IHttpBodyControlFeature>();
if (syncIOFeature != null)
{
    syncIOFeature.AllowSynchronousIO = true;
}

If you have trouble with TextWriters or other streams calling sync APIs in Dispose, call the new DisposeAsync API instead.

@kieronlanning

This comment has been minimized.

Copy link

@kieronlanning kieronlanning commented Feb 17, 2019

As-per aspnet/AspNetCore.Docs#10983, will this be back ported to 2.x's feature set?

Currently setting this to false in 2.2 breaks response compression.

@BrennanConroy BrennanConroy changed the title [Annoucement] AllowSynchronousIO disabled in all servers [Announcement] AllowSynchronousIO disabled in all servers Feb 17, 2019
@davidfowl

This comment has been minimized.

Copy link
Contributor

@davidfowl davidfowl commented Feb 17, 2019

No, it won't be back ported. It's a massive breaking change.

@Tratcher

This comment has been minimized.

Copy link
Contributor Author

@Tratcher Tratcher commented Feb 17, 2019

Note response compression was fixed in 3.0 using the new DisposeAsync API.

@GusBeare

This comment has been minimized.

Copy link

@GusBeare GusBeare commented Mar 7, 2019

I have been working on porting an angularJS app to .Net Core and just upgraded to Preview 3. After upgrade some of the client GET requests now blow with this error:

InvalidOperationException: Synchronous operations are disallowed. Call WriteAsync or set AllowSynchronousIO to true instead.

I don't get it because by default the angular requests are async and why do just some of them break?

I am using VS 2019 Preview and will be hosting IIS in process.

Any ideas?

@Tratcher

This comment has been minimized.

Copy link
Contributor Author

@Tratcher Tratcher commented Mar 7, 2019

@GusBeare what's the full stack trace?

@GusBeare

This comment has been minimized.

Copy link

@GusBeare GusBeare commented Mar 7, 2019

I get the following html in the response.

error.txt

@Tratcher

This comment has been minimized.

Copy link
Contributor Author

@Tratcher Tratcher commented Mar 7, 2019

@GusBeare let's take this to #8302

@Kaelum

This comment has been minimized.

Copy link

@Kaelum Kaelum commented May 17, 2019

This change in .NET 3.0 Preview 5 breaks the usage of XmlSerializer.Deserialize(Stream). Do we now need to change AllowSynchronousIO to true? Or should we be performing serialization differently? As stated by others, this is a significant break change over all previous versions of ASP.NET.

Reference dotnet/coreclr#24643

@Tratcher

This comment has been minimized.

Copy link
Contributor Author

@Tratcher Tratcher commented May 18, 2019

@Kaelum

This comment has been minimized.

Copy link

@Kaelum Kaelum commented May 29, 2019

@Tratcher the FileBufferingReadStream constructor above doesn't appear to be in the 3.0 Preview 5 libraries. Also, I have concerns that using FileBufferingReadStream might be too slow, if it is actually caching to the filesystem. We need to process a minimum of 12,000 requests/sec, which we do by keeping everything in memory. I tried using the BufferedReadStream object, but under the covers it is doing everything synchronously as well. This is what I have tried:

Works, but I have concerns with caching to the file system, and the constructor in your example doesn't exist:

using (FileBufferingReadStream readstream = new FileBufferingReadStream(request.Body, memoryThreshold, null, @"D:\Temp\", ArrayPool<byte>.Shared))
{
	await readstream.DrainAsync(CancellationToken.None);
	readstream.Seek(0L, SeekOrigin.Begin);

	XmlSerializer xmlSerializer = new XmlSerializer(typeof(XmlRequestBcap));
	xmlRequestBcap = xmlSerializer.Deserialize(readstream) as XmlRequestBcap;
}

Works, but probably non-performant:

using (StreamReader streamReader = new StreamReader(request.Body, Encoding.UTF8))
using (StringReader stringReader = new StringReader(await streamReader.ReadToEndAsync()))
{
	XmlSerializer xmlSerializer = new XmlSerializer(typeof(XmlRequestBcap));
	xmlRequestBcap = xmlSerializer.Deserialize(stringReader) as XmlRequestBcap;
}

Does not work, executes synchronously:

using (BufferedReadStream readstream = new BufferedReadStream(request.Body, memoryThreshold))
{
	if (!await readstream.EnsureBufferedAsync(CancellationToken.None))
	{
		throw new InvalidOperationException("Request body is empty.");
	}

	XmlSerializer xmlSerializer = new XmlSerializer(typeof(XmlRequestBcap));
	xmlRequestBcap = xmlSerializer.Deserialize(readstream) as XmlRequestBcap;
}

Variation of above that works, adding a MemoryStream, but is it safe and performant:

using (BufferedReadStream readstream = new BufferedReadStream(request.Body, memoryThreshold))
{
	if (!await readstream.EnsureBufferedAsync(CancellationToken.None))
	{
		throw new InvalidOperationException("Request body is empty.");
	}

	using (MemoryStream memoryStream = new MemoryStream(readstream.BufferedData.Array, readstream.BufferedData.Offset, readstream.BufferedData.Count, false))
	{
		XmlSerializer xmlSerializer = new XmlSerializer(typeof(XmlRequestBcap));
		xmlRequestBcap = xmlSerializer.Deserialize(memoryStream) as XmlRequestBcap;
	}
}

If FileBufferingReadStream is the most performant, and recommended for our case, what happened to the constructor that I see in your source reference? It is definitely not in the published 3.0 preview.

Thanks!

@davidfowl

This comment has been minimized.

Copy link
Contributor

@davidfowl davidfowl commented May 30, 2019

@Tratcher the FileBufferingReadStream constructor above doesn't appear to be in the 3.0 Preview 5 libraries. Also, I have concerns that using FileBufferingReadStream might be too slow, if it is actually caching to the filesystem. We need to process a minimum of 12,000 requests/sec, which we do by keeping everything in memory. I tried using the BufferedReadStream object, but under the covers it is doing everything synchronously as well. This is what I have tried:

It's there AFAIK.

@Kaelum

This comment has been minimized.

Copy link

@Kaelum Kaelum commented May 30, 2019

@davidfowl & @Tratcher I just verified that it is not in the SDK 3.0.100-preview5-011568 build. The referenced Microsoft.AspNetCore.WebUtilities assembly is pointing to:

C:\Program Files\dotnet\packs\Microsoft.AspNetCore.App.Ref\3.0.0-preview5-19227-01\ref\netcoreapp3.0\Microsoft.AspNetCore.WebUtilities.dll

When I disassemble that assembly, I can very definitely see only 4 constructors, with the 2 parameter constructor missing.

@Tratcher

This comment has been minimized.

Copy link
Contributor Author

@Tratcher Tratcher commented May 30, 2019

Here's the MVC PR: #9806
I think that was a preview6 change.

@FreyJim

This comment has been minimized.

Copy link

@FreyJim FreyJim commented Aug 12, 2019

@Tratcher
I have the same issue by using streams with moq framework. I changed already inside the moq setup to call the async methods and also added the section to the startup class.
services.Configure<TestServer>(options => { options.AllowSynchronousIO = });

But I still get the same error message:
System.InvalidOperationException: Synchronous operations are disallowed. Call WriteAsync or set AllowSynchronousIO to true.

Do I have to do something special by using the TestServer?

@Tratcher

This comment has been minimized.

Copy link
Contributor Author

@Tratcher Tratcher commented Aug 12, 2019

What's the stack trace?

@Tratcher

This comment has been minimized.

Copy link
Contributor Author

@Tratcher Tratcher commented Aug 12, 2019

How are you creating TestServer in your test? The old pattern was to do new TestServer(webhostbuilder); which wouldn't use DI to create an instance so services.Configure<TestServer>(options => { options.AllowSynchronousIO = }); wouldn't apply. If you are newing it up directly then you can directly set the option as well.

The new pattern looks like this and would work with DI options:
https://github.com/aspnet/AspNetCore/blob/8c02467b4a218df3b1b0a69bceb50f5b64f482b1/src/Hosting/TestHost/test/TestServerTests.cs#L53-L62

@benmccallum

This comment has been minimized.

Copy link
Contributor

@benmccallum benmccallum commented Oct 18, 2019

How would this look in the context of a WebApplicationFactory<Startup> test?

I'm doing the following in Startup under ConfigureServices, but I'm guessing the server is different in this scenario?

services.Configure<KestrelServerOptions>(options =>
{
    options.AllowSynchronousIO = true;
});
@benmccallum

This comment has been minimized.

Copy link
Contributor

@benmccallum benmccallum commented Oct 18, 2019

Sigh, how did I not see that. Was about to browse the source. Cheers again, sorry for the spam!

@Muchiachio

This comment has been minimized.

Copy link

@Muchiachio Muchiachio commented Oct 23, 2019

How to know which code part is throwing exception if we get this stack trace?
We don't really use our own text writers, only those passed to IHtmlContent.WriteTo method.

System.InvalidOperationException: Synchronous operations are disallowed. Call WriteAsync or set AllowSynchronousIO to true instead.
   at Microsoft.AspNetCore.Server.IIS.Core.HttpResponseStream.Write(Byte[] buffer, Int32 offset, Int32 count)
   at Microsoft.AspNetCore.Server.IIS.Core.WrappingStream.Write(Byte[] buffer, Int32 offset, Int32 count)
   at Microsoft.AspNetCore.WebUtilities.HttpResponseStreamWriter.FlushInternal(Boolean flushEncoder)
   at Microsoft.AspNetCore.WebUtilities.HttpResponseStreamWriter.Dispose(Boolean disposing)
   at System.IO.TextWriter.Dispose()
   at Microsoft.AspNetCore.Mvc.ViewFeatures.ViewExecutor.ExecuteAsync(ViewContext viewContext, String contentType, Nullable`1 statusCode)
   at Microsoft.AspNetCore.Mvc.ViewFeatures.ViewExecutor.ExecuteAsync(ActionContext actionContext, IView view, ViewDataDictionary viewData, ITempDataDictionary tempData, String contentType, Nullable`1 statusCode)
   at Microsoft.AspNetCore.Mvc.ViewFeatures.ViewResultExecutor.ExecuteAsync(ActionContext context, ViewResult result)
   at Microsoft.AspNetCore.Mvc.ViewResult.ExecuteResultAsync(ActionContext context)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeNextResultFilterAsync>g__Awaited|29_0[TFilter,TFilterAsync](ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Rethrow(ResultExecutedContextSealed context)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.ResultNext[TFilter,TFilterAsync](State& next, Scope& scope, Object& state, Boolean& isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.InvokeResultFilters()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeNextResourceFilter>g__Awaited|24_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Rethrow(ResourceExecutedContextSealed context)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.InvokeFilterPipelineAsync()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Logged|17_1(ResourceInvoker invoker)
   at Microsoft.AspNetCore.Routing.EndpointMiddleware.<Invoke>g__AwaitRequestTask|6_0(Endpoint endpoint, Task requestTask, ILogger logger)
   at Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddleware.Invoke(HttpContext context)
@Tratcher

This comment has been minimized.

Copy link
Contributor Author

@Tratcher Tratcher commented Oct 23, 2019

   at Microsoft.AspNetCore.WebUtilities.HttpResponseStreamWriter.FlushInternal(Boolean flushEncoder)
   at Microsoft.AspNetCore.WebUtilities.HttpResponseStreamWriter.Dispose(Boolean disposing)
   at System.IO.TextWriter.Dispose()
   at Microsoft.AspNetCore.Mvc.ViewFeatures.ViewExecutor.ExecuteAsync(ViewContext viewContext, String contentType, Nullable`1 statusCode)
   at Microsoft.AspNetCore.Mvc.ViewFeatures.ViewExecutor.ExecuteAsync(ActionContext actionContext, IView view, ViewDataDictionary viewData, ITempDataDictionary tempData, String contentType, Nullable`1 statusCode)

@NTaylorMullen ?
Looks like an MVC bug.

@Muchiachio

This comment has been minimized.

Copy link

@Muchiachio Muchiachio commented Oct 23, 2019

I pulled all the sources and found the problem. A 3rd party library had a bug which threw an exception in a partial view rendered by Html.Partial (no async to sync code used). It's still strange that it was caught as a sync IO problem only when exception was thrown. I can share a repro in a separate issue if @NTaylorMullen is interested.

@Tratcher

This comment has been minimized.

Copy link
Contributor Author

@Tratcher Tratcher commented Oct 23, 2019

@Muchiachio please open a new issue for this scenario, it sounds like we need to fix something in MVC.

@NTaylorMullen

This comment has been minimized.

Copy link
Contributor

@NTaylorMullen NTaylorMullen commented Oct 23, 2019

I pulled all the sources and found the problem. A 3rd party library had a bug which threw an exception in a partial view rendered by Html.Partial (no async to sync code used). It's still strange that it was caught as a sync IO problem only when exception was thrown. I can share a repro in a separate issue if @NTaylorMullen is interested.

@Muchiachio Filing an issue with a repro would be a great start!

dracan pushed a commit to dracan/danclarkeblog that referenced this issue Oct 24, 2019
Dan Clarke
Method now has to be async:
dotnet/aspnetcore#7644
@RomanTymchyshyn

This comment has been minimized.

Copy link

@RomanTymchyshyn RomanTymchyshyn commented Oct 25, 2019

@MaddoxDevelopment if you have a repro then send me the details with steps, that shouldn't happen.

@MaddoxDevelopment , @davidfowl , so did you figured out what the problem was in that case?

I have a similar problem, I use .AddNewtonsoftJson, but then receiving "Synchronous operations are disallowed. Call ReadAsync or set AllowSynchronousIO to true instead.".
You can see in the stack trace, that NewtonsoftJsonInputFormatter gets called.

StackTrace:

       at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpRequestStream.Read(Byte[] buffer, Int32 offset, Int32 count)
       at Microsoft.AspNetCore.WebUtilities.FileBufferingReadStream.Read(Byte[] buffer, Int32 offset, Int32 count)
       at Microsoft.AspNetCore.WebUtilities.HttpRequestStreamReader.ReadIntoBuffer()
       at Microsoft.AspNetCore.WebUtilities.HttpRequestStreamReader.Read(Char[] buffer, Int32 index, Int32 count)
       at Newtonsoft.Json.JsonTextReader.ReadData(Boolean append, Int32 charsRequired)
       at Newtonsoft.Json.JsonTextReader.ParseValue()
       at Newtonsoft.Json.JsonReader.ReadAndMoveToContent()
       at Newtonsoft.Json.JsonReader.ReadForType(JsonContract contract, Boolean hasConverter)
       at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Deserialize(JsonReader reader, Type objectType, Boolean checkAdditionalContent)
       --- End of stack trace from previous location where exception was thrown ---
       
       at Microsoft.AspNetCore.Mvc.Formatters.NewtonsoftJsonInputFormatter.ReadRequestBodyAsync(InputFormatterContext context, Encoding encoding)
       at Microsoft.AspNetCore.Mvc.ModelBinding.Binders.BodyModelBinder.BindModelAsync(ModelBindingContext bindingContext)
       at Microsoft.AspNetCore.Mvc.ModelBinding.ParameterBinder.BindModelAsync(ActionContext actionContext, IModelBinder modelBinder, IValueProvider valueProvider, ParameterDescriptor parameter, ModelMetadata metadata, Object value)
       at Microsoft.AspNetCore.Mvc.Controllers.ControllerBinderDelegateProvider.<>c__DisplayClass0_0.<<CreateBinderDelegate>g__Bind|0>d.MoveNext()
       --- End of stack trace from previous location where exception was thrown ---
       
       at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeInnerFilterAsync>g__Awaited|13_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
       at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeNextExceptionFilterAsync>g__Awaited|25_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)

UPD: Maybe will be useful for somebody. The problem was that I used logging middleware with a call to Request.EnableBuffering().

@davidfowl

This comment has been minimized.

Copy link
Contributor

@davidfowl davidfowl commented Oct 25, 2019

@MaddoxDevelopment fixed in 3.1 #14396 cc @pranavkm

@olaj

This comment has been minimized.

Copy link

@olaj olaj commented Oct 29, 2019

@benmccallum @Tratcher About the TestServer implementation. I have tried to set that AllowSynchronousIO property on various places but i only get a Stack Overflow exception in the program.cs, or i get the error "Synchronous operations are disallowed. Call ReadAsync or set AllowSynchronousIO to true instead.". Is there any example somewhere on how to actually set this property? Where do i set it?

My setup is similar to what they do in this tutorial. https://docs.microsoft.com/en-us/aspnet/core/test/integration-tests?view=aspnetcore-3.0

@davidfowl

This comment has been minimized.

Copy link
Contributor

@davidfowl davidfowl commented Oct 29, 2019

Stackoverflow? Can you provide a repo?

@Tratcher

This comment has been minimized.

Copy link
Contributor Author

@Tratcher Tratcher commented Oct 29, 2019

@olaj did you try the sample I gave above #7644 (comment)?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
You can’t perform that action at this time.