Gradual warm-up of recovered destination

[alnikola]

An unhealthy destination that was excluded from the available destination list can eventually recover and start receiving traffic again. By default, it will get the same load as it was before the failure. It is a fast way to recover the full system capacity that is justified in case the root cause of the failure is fixed for good. However, if the issue is not fully solved and the destination is still unstable, an instant increase of the load can overwhelm it and bring down again.

To avoid such outcome, it's proposed to add a gradual destination warm-up strategy. This can be achieved by assigning to all destinations the new weight value in range (0, 1] controlling the probability of a destination being chosen by the load balancer. By default all destination gets 1, but if one failed and then recovered, its weight gets reset to 0 and then gradually increased back to 1.

[karelz]

Triage: It will be part of more general weight design, which should ideally happen in 1.0, unless it turns out to be too complicated.