Rate limit: Implement token buckets#6983
Rate limit: Implement token buckets#6983cplaursen merged 2 commits intoxapi-project:feature/throttling2from
Conversation
cplaursen
force-pushed
the
feature/throttling
branch
from
7cfb013 to
69ee866
Compare
cplaursen
force-pushed
the
feature/throttling
branch
from
059e64d to
fa65aaf
Compare
lindig
left a comment
lindig
left a comment
There was a problem hiding this comment.
This looks good and I assume is well tested. At the same time, the execution is difficult to observe because there is very little logging outside the logging an operation does itself. So we don't see how long something stays in the queue before it is started or how long an execution takes - when this information is probably available.
|
|
||
| (* Block and execute on the same thread *) | ||
| let submit_sync bucket_data ~callback amount = | ||
| check_not_terminated bucket_data.should_terminate ; |
There was a problem hiding this comment.
Is there an easy way to measure the duration an execution took and to log it? This way we could spot executions that take too long.
There was a problem hiding this comment.
There is a Xapi_stats module that could be used. Longer term the opentelemetry metrics feature could be used to expose more counters.
Meanwhile we already have opentelemetry spans that contain high-resolution begin/end timestamps, so when distributed tracing is enabled, this information can be extracted from there.
I think it'd be useful to create an opentelemetry span whenever rate-limiting kicks in, to make this more visible though. Perhaps @GabrielBuica could help with picking a good place to add distributed tracing spans in the rate limiting code.
We have a 'thread-diagnostics' CLI command that can print what each thread does. It also has some very minimal implementation to report what each thread is waiting on (see the named mutex implementation). The module to look at is Locking_helpers. |
robhoes
left a comment
robhoes
left a comment
There was a problem hiding this comment.
Generally looks good. Just one comment.
Token buckets form the core of the XAPI rate limiter. Token buckets hold tokens, which are refilled over time according to their refill rate up to a maximum. Each client making calls to XAPI will have one token bucket associated; each call takes tokens out of the bucket, and delays will be enforced such that any requests made to an empty bucket have to wait until the bucket has been refilled. The token bucket library implements functions for token bucket creation and thread-safe token consumption. Refills are calculated only when the bucket is observed by tracking a last_refill timestamp. Signed-off-by: Christian Pardillo Laursen <christian.pardillolaursen@citrix.com>
Token buckets form the basis of the rate limiting, but on their own they lack a way of enabling fairness when waiting. For this, we add a separate queuing mechanism which registers callbacks for requests that are waiting on a token bucket to refill. A worker thread is tasked with awakening requests as enough tokens are placed back in the bucket. We expose an interface for submitting requests to a queue, both in synchronous and asynchronous mode. Signed-off-by: Christian Pardillo Laursen <christian.pardillolaursen@citrix.com>
cplaursen
force-pushed
the
feature/throttling
branch
from
f30e754 to
7e072bc
Compare
4 participants
Token buckets form the core of the XAPI rate limiter. Token buckets hold tokens, which are refilled over time according to their refill rate up to a maximum.
Each client making calls to XAPI will have one token bucket associated; each call takes tokens out of the bucket, and delays will be enforced such that any requests made to an empty bucket have to wait until the bucket has been refilled.
The token bucket library implements functions for token bucket creation and thread-safe token consumption. Refills are calculated only when the bucket is observed by tracking a last_refill timestamp.