Contains the primitives and utilities used to rate-limit / throttle Java applications, and a CircuitBreaker implementation.
Inspired by reading Cal Henderson's "Building Scalable Web Sites" which talks briefly about this, and having been on the receiving end of a kicking from search engines, I wanted to have a simple way of determining whether to bother processing requests and stop consuming server resources in a graceful way, rather than grinding to a halt.
Each time a request comes in, we log the time. If it hasn't been a certain duration since the last request, then abort with a rate-limiting error.
key = create_key(request)
entry = gate.get_entry(key)
if (entry)
response.set_status(SERVICE_UNAVAILABLE)
return
end
entry = create_entry(expires => '5s')
gate.put_entry(key, entry)
...
We define a duration and an acceptable number of requests to be serviced in that time. Each time a request comes in, we look up the number of calls made in the current period. If it is at or above the limit, then abort with a rate-limiting error, otherwise increment the counter and service the request.
key = create_key(request)
entry = gate.get_entry(key)
if (entry.count >= ALLOWED_PER_PERIOD)
response.set_status(SERVICE_UNAVAILABLE)
return
end
entry.count.increment()
...
From this description, it can be seen that Next Service Slot is essentially Fixed Bucket with a max size of 1 and an appropriate service period.
Similar to a Fixed Bucket, except that rather than aborting, we block until the end of the current time period upon which the bucket counter is decremented / completely emptied and then we service the request.
Hardest to implement, has the disadvantage that it will tie up a request-handling thread (which may cause upstream services to timeout / retry) but may be a good solution in other contexts.
key = create_key(request)
entry = gate.get_entry(key)
if (entry.count >= ALLOWED_PER_PERIOD)
entry.wait()
end
entry.count.increment()
...
There is some overlap in the intention of this library with the Circuit Breaker approach described by Michael Nygard in his excellent book "Release It!"; I've done some work to add support for that as well. We've been running it in production for a year and it works well for our purposes.
Please see the tests for details as to how to use it.