Limiting concurrent requests from expect handler

[twisteroidambassador]

I'm writing an API endpoint that potentially allows large POST uploads, and need to limit the number of requests in progress at the same time. So I have something like this:

async def handler(request):
    if request.app['requests_in_progress'] >= request.app['max_concurrent_requests']:
        raise HTTPTooManyRequests()
    request.app['requests_in_progress'] += 1
    try:
        # handle the request
    finally:
        request.app['requests_in_progress'] -= 1

Now, I want to reject the request earlier in the Expect: handler. My question is:

Is it theoretically possible that, for a certain request, the expect handler does run, but then the main handler does not run? For example, maybe if the client disconnects at just the right moment?

If this scenario is guaranteed impossible, then I can test and increment counter in the expect handler, set a flag on the request instance, and skip incrementing in the main handler, thus supporting requests with and without Expect header. On the other hand, if the scenario is possible in theory, then by Murphy's law it will happen in production and increment the counter all the way without ever decrementing it, leading to my service rejecting all requests.

Please also enlighten me if there's a much better way than the above to handle concurrency limits.

[Dreamsorcerer]

I'm not too sure, but I suspect the answer is no. Middlewares are going to be run, so there is likely plenty of time in middlewares for the task to get cancelled. It's certainly not documented behaviour, so I wouldn't rely on it even if you can prove that it won't cancel with the current code. I'd suggest just implementing it in a middleware (make it the first middleware that is run), that's what they're there for.

Additionally, the expect handler will only be called if the expect header is present, so not very useful if you want this as some kind of security feature.

If the idea is to allow the client to avoid sending the body if it is going to be rejected, I'd suggest doing a check without incrementing the count in the expect handler, then increment it in a middleware. Maybe there is a very rare race condition where the server says OK, and then rejects the body, but it shouldn't be frequent enough to be a problem. Maybe there's also something clever you can come up with to ensure that it will always allows the connection after doing an expect check (like adding a flag to the request that bypasses the count check).

[twisteroidambassador]

Re: middlewares, are they a good fit if I only want to limit concurrency on a single endpoint / path?

The idea is indeed to tell the client about the rejection before it sends the large body. I thought about doing something clever like attaching a weakref finalizer to the request object, but eventually decided against it.

doing a check without incrementing the count in the expect handler, then increment it in a middleware. Maybe there is a very rare race condition where the server says OK, and then rejects the body, but it shouldn't be frequent enough to be a problem.

This sounds like the most sensible thing to do. Worst case, client gets a 100 Continue, wastes some bytes sending the body before being told 429 Too Many Requests.

[Dreamsorcerer]

For a single endpoint, then a decorator is probably the cleanest way.

[Dreamsorcerer]

Maybe it's also possible to avoid the entire counting problem altogether, by just looking at the number of connections or tasks currently in use. e.g. https://docs.aiohttp.org/en/stable/web_reference.html?highlight=connections#aiohttp.web.Server.connections
Although I don't think there's an easy way to get a reference to the Server from an Application.

[mshandrovskiy]

Have you considered not doing this at the application level at all? There are good rate limiting options already built-in nginx or other reverse proxies of your choice.

[Dreamsorcerer]

This is probably the most sensible answer. I'd personally look at these options first if I was doing it myself, nginx has something built-in and then there are WAFs as well.