[Epic] Request Throttling and Queuing Middleware

[analogrelay]

Optimistically putting this in 3.1, but no committments as to when exactly this ships. It won't be part of 3.0.

This is tracking work to add a Request Throttling and Queuing middleware (or possibly middlewares) to ASP.NET Core. The purpose of this is to enable app developers to restrict the number of concurrently-running requests in their application in order to better manage resources.

Currently, the only queuing that occurs in ASP.NET Core applications is through use of the thread-pool queue, which queues completions of async operations (i.e. async/await) and other thread-pool work items. This can be difficult to manage and measure. For some applications, the introduction of a concurrent request limit can mitigate issues caused by problematic async patterns (like "sync-over-async" where blocking calls are done in async code paths). The "Right Thing" to do in most of these cases is to unravel these "sync-over-async" calls into a true async/await pattern, but sometimes that's infeasible or too costly. Limiting concurrent requests conserves these resources.

When there is a concurrent request limit, it is also desirable to "queue" requests that come in over this limit rather than turning them away immediately. This also provides a clear metric to use to measure the impact of load on your system (the number of requests waiting in the queue).

Our goal is to build a middleware that limits the number of concurrently executing requests in the remainder of the pipeline. The middleware will also queue requests beyond that limit until there is space. Finally, if the limit and the queue are exhausted, it will reject the request immediately (usually by emitting a 503 HTTP status code).

We do believe that components of this middleware may be useful for "throttling" as well. Throttling includes scenarios like API Rate Limits, where requests are tracked against certain per-user limits and rejected when they exceed that limit. This is beyond the current scope of the middleware but may serve as a stretch goal.

NOTE: This is our summer intern project, so bear with us as we work through it. We've shipped many of our intern projects as production code in future releases and intend to do so with this one as well!

Child Items:

• Basic middleware structure.
• Concurrent request limit.
• Queuing requests beyond the limit.
• Microbenchmarks for the Request Queuing middleware #10703 Microbenchmarks.
• Measure overhead of Request Queuing middleware in Plaintext benchmark #10704 Measure the overhead of queuing in the Plaintext benchmark.
• Request Queuing middleware should emit Event Counters #10705 Add Event Counters and use dotnet-counter tool to view them.
• Add benchmark scenarios to illustrate where Request Queuing benefits #11339 Add benchmark scenarios to illustrate where Request Queuing benefits
• Performance exploration in Request Queuing middleware #10706 Optimizations based on benchmarks. Use a profiler to explore further.
• Add OnRejected callback to Request Queuing middleware to allow user code to handle rejections #10707 OnRejected callback to allow user code to handle requests rejected due to insufficient queue space.
• Multiple queues and queue assignment. (Func<HttpContext, Queue>, 💪 stretch goal)

[rynowak]

Can we get some cool info about what this is and how it be what it do?

[halter73]

The cliff's note's versions is that this middleware is designed to prevent threadpool starvation just like the System.Web request queue. It will be able to enforce a configurable concurrent request limit and start queuing new requests once the concurrent request limit is reached. Of course, the number of requests that can be queued at once is also configurable. If the request queue fills up, requests will be rejected with a generic 503 or custom handler.

There's still discussion on how useful it might be to allow certain (e.g. admin) requests to bypass the main request queue. That's what "Multiple queues and queue assignment. (Func<HttpContext, Queue>)" is referring to. Whether we just blog about this middleware, include it in some templates, or something else is still up in the air. I doubt it's something we'd be able to turn on by default.

[halter73]

Of course, things have changed from the early days of System.Web. Most importantly, a request doesn't necessarily block a thread, but sometimes they do.

[rynowak]

Of course, things have changed from the early days of System.Web. Most importantly, a request doesn't necessarily block a thread, but sometimes they do.

This is also something I deffo remember discussing in the early days of K. For whatever reason we didn't built it in, but there was discussion about making this an MVC filter - (middlware is a better choice).

There's still discussion on how useful it might be to allow certain (e.g. admin) requests to bypass the main request queue. That's what "Multiple queues and queue assignment. (Func<HttpContext, Queue>)" is referring to. Whether we just blog about this middleware, include it in some templates, or something else is still up in the air. I doubt it's something we'd be able to turn on by default.

If you end up having this feature, I'd suggest showing a sample that does that based on endpoints and metadata. Then you can slap an attribute on a method to configure the policy.

You might also want to write a generic [QueueName(...)] attribute that can be used with routing to drive all of this. Attributes can be applied to hubs/grpc/controllers/pages, and can also be used with routerware (middleware pipeline + route + metadata like auth policies).