Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
502.3 bad gateway errors couple of hours after deployment #323
I deployed an asp.net core 1.0 webapi service to Azure web app in Standard tier. It works fine for a couple of hours. But starts throwing 502.3 bad gateway errors after that. Restarting the web app again fixes it. But it again behaves the same after some time. We deploy the service from VSTS. Please help
I'm also having severe issues with an ASP.NET Core 1.1 app running .NET Core on Azure. Everything will be running fine for hours, but then we'll suddenly have a huge spike in threads being used and the server will throw 502 errors.
Here's a screenshot of a good instance and a bad instance:
Here's the graph of threads being used by dotnet.exe
And the graph of what happens to our site when this occurs:
I took a memory dump of the w3wp and dotnet.exe processes if that will help diagnose the issues. I've read through most of the other threads linked but none of them has a definite solution that I can implement.
The site basically takes in form posts, transforms the data and posts messages onto Azure Service Bus. It also pulls back some data from Redis and returns it. Overall, it's quite basic.
I've seen references to some things being fixed in 1.1.1, but I haven't seen an ETA on when that will be released. I'm open to suggestions on immediate fixes.
@BrianVallelung @nemenos The increasing number of threads might have been caused by a deadlock in Kestrel. The best way to verify this is to to take a memory dump and use VS's Debug Managed Memory Feature to take a look at the parallel stacks windows to see if it contains the same stacks as the screenshot in the linked Kestrel issue.
If the stacks do look the same. You can work around this issue by not using the ASP.NET Core to serve static files, and serve files directly using IIS as suggested here. This works, because in a typical app, the only thing that passes a CancellationToken to Response.WriteAsync() is the static file middleware. The deadlock will be fixed in 1.1.1 and in a 1.0.x backport which will be released soon.