Stats for Ring Buffer

[bRhoa]

While investigating why clients were not respecting the client side timeout, we learned that the call to put a command on the ring buffer may block indefinitely while queuing for a ring slot, as the context timeout is only checked when the command is dequeued.

We think that it would be useful to have a metric to track how many requests are getting queued and/or for how long, as they may be interesting for determining whether we are bottlenecking on too few connections. We would be willing to contribute if you think the idea is worth investing in.

[rueian]

Hi @bRhoa,

Thank you for pointing it out. Yes, the PutXXX can hang indefinitely. Ultimately, I think it should respect context deadlines as well.

I agree improving observability of the ring queue is a good idea. Metrics for how many inflight request are waiting for the ring queue can be easy and cheap. But I am afraid how long do those request are waiting can be expensive. Can we have a sampling or approximate approach for latencies?

[bRhoa]

We could probably start with the count of waiting requests. That alone would be very valuable.

The latency would be in the hot path, so maybe some sort of sampling aggregator that is updated on a fixed interval. I don't have a good answer for what that would look like yet.