Request hedging

[jakedt]

From the paper:

For calls to Spanner and to the Leop- ard index we rely on request hedging [16] (i.e. we send the same request to multiple servers, use whichever response comes back first, and cancel the other requests). To reduce round-trip times, we try to place at least two replicas of these backend services in every geographical region where we have Zanzibar servers. To avoid unnecessarily multiply- ing load, we first send one request and defer sending hedged requests until the initial request is known to be slow.

To determine the appropriate hedging delay threshold, each server maintains a delay estimator that dynamically computes an Nth percentile latency based on recent mea- surements. This mechanism allows us to limit the additional traffic incurred by hedging to a small fraction of total traffic.

Fixes #19

[jzelinskie]

I wonder if it's overkill to pull this out into its own type that has thread-safe methods like Quantile().

[jakedt]

I'm honestly not sure, we would still need a lock around any mutations to the slice, and we're reading from the digests just as often as we are writing to them.

[josephschorr]

But if we move the writes into their own routine (see below), we could in theory either switch to a RWLock or, alternatively, have the "write" call also store the current first quantile value, and have reads just use whatever is stored?

[jakedt]

Is there a change to make here?

[josephschorr]

I'd see if we could avoid the lock entirely on the read, only take it on the write (ideally) and do so asynchronously from the critical path

[josephschorr]

Place the compression into a const?

[josephschorr]

Maybe put a comment here? I know Go doesn't auto-fallthrough, but it first read that way

[jakedt]

This is a select not a switch...

[josephschorr]

I know, but it still looks that way at first read through

[jzelinskie]

idk about that and even for most cases you're unlikely to be even want to use fallthrough because you can do multiple cases separated by commas (e.g. case This == false, That == false:).

[josephschorr]

Does order matter at all here? If not, I'd say we update the digest in a goroutine to prevent this from being on the critical call path

[jakedt]

They take microseconds to query and update according to the original author of the algorithm.

[josephschorr]

But if we move the writes into their own routine (see below), we could in theory either switch to a RWLock or, alternatively, have the "write" call also store the current first quantile value, and have reads just use whatever is stored?

[jzelinskie]

you should just use promauto