Original vs hedged request metrics

[cristaloleg]

Fixes #42

[dannykopping]

Exactly how I was planning to implement it 🙂
LGTM!

[cristaloleg]

Thanks, then I will add tests to cover this case and release a new version soon.

[dannykopping]

Really appreciate it @cristaloleg 🙌

[cristaloleg]

Tests are added @dannykopping @joe-elliott . Not sure about thread above, IMO having 2 counters is much simpler and costs nothing, WDYT ?

I will refactor tests in the next PR, looks too wordy now.

[dannykopping]

🎉🎉🎉

[dannykopping]

I tried to integrate this, but long story short we have many hedged clients but only register the metrics once; this makes it difficult to use the stats because one client might have 0 for HedgedRequestWins() and another might have some other value - which one would we use to update the metrics? The answer is: without major refactoring, we can't.

I solved this by using the return value of the roundtrip:

func (rt *limitedHedgingRoundTripper) RoundTrip(req *http.Request) (*http.Response, error) {
	isHedged := hedgedhttp.IsHedgedRequest(req)
	if isHedged {
		if !rt.limiter.Allow() {
			totalRateLimitedHedgeRequests.Inc()
			return nil, ErrTooManyHedgeRequests
		}
		totalHedgeRequests.Inc()
	}

	resp, err := rt.next.RoundTrip(req)
	if err == nil {
		if isHedged {
			requestsWon.WithLabelValues("hedged").Inc()
		} else {
			requestsWon.WithLabelValues("original").Inc()
		}
	}

	return resp, err
}

Edit: here's the PR grafana/loki#10281

[cristaloleg]

Thanks for the update! So...as I understood it's not library's problem but only a case that you've many hedged clients in 1 app. Am I right?

BTW, any numbers how many wons? :D (such info is probably under NDA but I decided to try :D )

[dannykopping]

So...as I understood it's not library's problem but only a case that you've many hedged clients in 1 app. Am I right?

That's correct!

BTW, any numbers how many wons? :D (such info is probably under NDA but I decided to try :D )

I'll update you once I roll this out to production next week 🙂 I'm sure I can send a % of hedging effectiveness with no problems.

[dannykopping]

This is from a 12 hour range in our pre-prod environment. Seems like we're averaging about a 20% effectiveness rate (i.e. only 1 of every 5 hedged requests wins vs the original).

Do you have a good intuition about what this rate should be in order to justify the extra expense?

[cristaloleg]

Do you have a good intuition about what this rate should be in order to justify the extra expense?

Sorry, no real formulas, someone with a better probability theory background should comment on that.

But the intuition suggests that it should not be high (or even lower than that). Request hedging is only about tail latency, so basically 1-5% of the requests.

I don't think there is are exact numbers for everyone. Sleep between calls, amount of hedged calls and success rate as a result heavily depends on the systems and it's behaviour. The only thing I can really suggest is to play with the numbers.

Probably the target latency should be a const (so, SLO) and after that tweaking sleeps & amount should be adjusted to minimise numbers of calls but keeping latency at the desired level. My 2c.

CC: @storozhukBM