-
Notifications
You must be signed in to change notification settings - Fork 10
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Rate limit hedge requests #18
Comments
I'm thinking that having a ceiling is going to be more useful in practice than a %. Let's say we hedge 10%, and have a retry ( If the upstream service we're hedging requests for slows down drastically, we'll be re-issuing all 10% of our requests x 3 (1 initial request and 2 retries), which will triple our latency in 10% of requests. If we're hedging more than a couple hundred requests a minute, that's probably pathological on the upstream service's side or we're being too aggressive. |
How about putting an actual rate limit on hedged requests, like "you can issue hedged requests up to 10 times per second". |
Hi almost all Grafana team 😂 yeah, great idea, like it. I had something on my mind long time ago but left for the future. But later left untouched 'cause now it's possible to substitute rate limited RoundTripper (https://github.com/cristalhq/hedgedhttp/blob/main/hedged.go#L53) which will take care not to overload targets.
In other words: passing |
We actually just want to rate limit hedge requests and not other requests, so I don't think that works since it will rate limit everything right ? |
A-ha, got it. Well, yeah, probably there is no such way right now. Unless it's possible to make something with request so the I also thought about something else: provide a param to So:
Looks like this should solve the problem started above and probably will make observability easier. Objections? :) |
Sounds like it would work perfectly. |
One more idea discussed with @storozhukBM in priv: help to distinguish 1st and following requests via context. Maybe We pay small-ish price creating context with value but the underlying RoundTripper can decide what to do with a request, based on the result of Ah, and also tests can be simpler, again due to 1 helper. WDYT? |
Made a simple PR with |
I just realized that once a hedged request is fired it;s already too late. What we want actually is to control if we should hedged or not. So while this help for instrumentation, I don't think it works for rate limiting. |
Well, it's created and passed to a RoundTripper but only RoundTripper decides to pass it further and send over a network. However we(you?) might want to catch it earlier.
From another perspective: what is the problem with context approach? price of a request creation is small, metrics aren't skewed (https://github.com/cristalhq/hedgedhttp/blob/main/hedged.go#L117 is correct, request is failed due to rate limit), rate limit is correct and we can create such Am I missing something? |
We could definitively fails the hedge request what is the behaviour for the original requests ? |
The same as before - nothing happens. |
ok so if a hedge request returns an error it's discarded. |
Yes, exactly. |
Then I think we're good to close this once the PR is merged. I close the Prometheus one because your argument were good enough for me. |
Released https://github.com/cristalhq/hedgedhttp/releases/tag/v0.6.2 |
Thank you for the prompt release. |
Tripping over this. Thanks @cristaloleg! Clearly we love this project over at Grafana. |
@annanay25 super glad to hear 😉 ❤️ |
New release, examples and better README (and a bit better CI) https://github.com/cristalhq/hedgedhttp/releases/tag/v0.6.3 (diff v0.6.2...v0.6.3) |
Hello there,
What do we think about adding a way to rate limits hedged requests ?
For example if we suddenly seems to hedge all request because the tail latency changed, we may actually worsen the problem.
I suggest we add a percentage parameter which tells how often we can hedge request based on actual requested throughput.
For example if the value is 10%, then we can only hedge 1 request out of 10. Default could be 100%, all request can be hedged.
What do you think ?
The text was updated successfully, but these errors were encountered: