Limiting incoming requests queue

[kinnalru]

Is it possible to limit incoming request queue by queue length or timeout?
Something like max_request_queue_size and max_request_queue_time from Phusion Passenger?

[ioquatix]

At this time, there is no such feature and I agree it's a deficiency. I am planning to address this (configurable load balancing, rate limiting, etc), but it may be a paid feature. How do you feel about that?

[kinnalru]

@ioquatix

it will be great if falcon starts bringing you money. But maintaining both versions (enterprise and free) is difficult.

[slewsys]

At this time, there is no such feature and I agree it's a deficiency. I am planning to address this (configurable load balancing, rate limiting, etc), but it may be a paid feature. How do you feel about that?

Nginx offers rate limiting and load balancing. You might have more success offering a hosting service, rather than charging for tooling.

[platbr]

Is it possible to limit incoming request queue by queue length or timeout? Something like max_request_queue_size and max_request_queue_time from Phusion Passenger?
HAProxy offers a queue feature using maxconn.
NGINX open source doesn't have a queue feature, only Plus version.

[platbr]

Thoughts on Queueing Requests/Scaling

We moved from Puma to Falcon last week because it is much easier to handle scaling without needing to worry about the number of necessary threads or how CPU resources are allocated. We have five applications working together to provide all the functionality of our website (we sell tickets).

Before Falcon, we scaled based on Puma's backlog and CPU usage, but now we scale solely based on CPU. This change allows us to ensure that CPU resources are fully utilized. However, we had to drastically increase the amount of memory, which I assume is because Falcon handles more requests simultaneously, leading to this behavior.

Since Falcon doesn't have a request queue, I’ve been considering placing it behind HAProxy. However, this might reintroduce the same issue we faced with Puma, where we couldn’t fully utilize available CPU. On the other hand, it could help limit the number of simultaneous requests.

I believe a good approach for queueing requests could be to base it on available CPU. Is it possible to control the queue this way? Can available CPU be monitored effectively in a shared cloud environment , Kubernetes, inside a container, etc?

@ioquatix , Do you think this would be a good approach?

[ioquatix]

It is now possible to do this, see https://github.com/socketry/falcon/blob/main/examples/limited/readme.md for details. Right now, it's something you must integrate yourself, but in the future we may adopt it as a standard feature.