-
Notifications
You must be signed in to change notification settings - Fork 9.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Kestrel execution timeout support #10079
Comments
ASP.NET 4.x implemented this using Thread Abort which only worked because it assumed the request was executed by a single thread (or tracked by the execution context?). There is no plan to bring back Thread Aborts nor would they be likely to work very well now that Core is fundamentally asynchronous. |
Thanks @Tratcher, I understand that, |
This isn't a web server specific problem, it could happen in any application. It's quite difficult for the application to self-diagnose and distinguish between an expensive operation and a broken one, let alone recover from it in such a way that the application is stable afterwards. For this class of error I think you're better off using external monitoring and health checks. If the application ceases to respond in an appropriate manner then it can be terminated and restarted by an external agent and return to a stable state. |
@Tratcher Would you like me to make such a remark (your 2nd para) in the vicinity of ANCM request timeout coverage for in-proc in the topic? If not, I'll close the doc issue. |
No, ANCM doesn't really handle this. @glennc does this make sense in the context of health checks? |
btw - IIRC, it's the only place this type of timeout is covered, so that's why I thought it might be the best spot for a passing remark. Alternatively, we could surface it in a very short section of the server's topic. https://docs.microsoft.com/aspnet/core/fundamentals/servers/ The prob with Health Checks is that devs probably won't look there for this ... and search may leave them hanging. |
There's no way to forcibly terminate .NET Closing this as we have no plans to introduce a system to forcibly terminate request processes. |
Thank you everyone, using (var TokenSource = System.Threading.CancellationTokenSource.CreateLinkedTokenSource(Context.RequestAborted)) |
Hi,
We've been facing some high CPU issues on our ASP.NET core 2.2 production environment for the past few weeks. The only way to "revive" our instances, was to restart Kestrel (or the server instance of course).
Following some hard time, we ended up finding some very rare scenario, in which our application was going through an infinite loop, which Kestrel never stopped processing.
This means any single request hitting this bug, created a 100% CPU usage task on our server instance, which never goes away...
We tried using the "RequestTimeout" setting on web.config (after switching to OutOfProcess hosting model), which returned an error to the client ("The specified CGI application encountered an error and the server terminated the process"), however, the server kept processing the request infinitely.
I believe the ExecutionTimeout which never made it to Kestrel on ASP.NET core might be a good solution for error handling on the server level.
For additional regarding ExecutionTimeout information, please see @Tratcher and @benaadams review and remarks here: aspnet/KestrelHttpServer#485, aspnet/KestrelHttpServer#611.
To re-produce the issue, a simple "while (true)" should be called during the request processing. We placed it on the view for simplicity.
Obviously all of our code is using the RequestAborted cancellation token when possible, yet in the case we mentioned, we had a complex code which shouldn't have been a long running CPU task, but ended up acting as an infinite loop.
We also tried creating a Middleware for limited the execution time, it didn't much good either, as we couldn't a way to "kill" the running task. Perhaps it could be done?
Tagging for future documentation purposes: dotnet/AspNetCore.Docs#12330
Thank you,
Effy
The text was updated successfully, but these errors were encountered: