"Degraded" health check

[snowp]

Right now Envoy health checks are binary yes/no that determine whether a host should receive traffic. A possible extension to this would be to provide additional information in the http health check response that allows the downstream to reprioritize the host.

For instance, you can imagine an upstream responding with x-envoy-health-degraded when the host is able to serve most traffic but say a Redis shard is down. This would make the downstream prefer other hosts that aren't declaring themselves degraded, presumably because they have a healthy Redis shard.

Floating this idea since it's something we have in our legacy RPC system and internal users seem to like it.

[mattklein123]

@snowp this request has come up in different forms multiple times. The "devil is in the details" on this one as it can get quite complicated, scary, and error prone. If you feel like doing a design proposal that would be great.

[snowp]

I have a few high level ideas on how to accomplish this:

• Reuse priorities: allow health checks to move hosts between priorities. You would shift a degraded host's priority by N where N is the number of priorities. P0 becomes PN, P1 becomes P(N-1). This maintains the relative priority between degraded hosts and reuses all of the existing LB mechanisms like spillover etc.
• Multiple LBs: We could maintain several LBs for different ties of hosts (healthy, degraded, maybe several levels of degraded?) and attempt to select from them in order. This is a much larger change so I'm not sure if it's worth it. It also gets tricky because the existing LB already has fallbacks for when there are no healthy hosts.
• Support degraded hosts in LB: Filter out degraded hosts during host updates. Similar to how unhealthy hosts are partitioned from healthy hosts, we could partition out the degraded hosts and maintain that state within the LB and add new rules for how degraded/unhealthy hosts interact (would degraded affect spillover? locality weight? etc.)

The first point is how we're implementing this on the control plane level and it's been working decently well. The fact that it doesn't complicate the routing logic by introducing new concepts in Envoy is a nice property, but I'm not sure if frequent priority moves will be problematic.

Happy to consider other options too, these are just some approaches.

[snowp]

Here's a more concrete suggestion based on the first high level approach in the previous comment:

Add a degraded_priority_load_ field to the LB which holds the load given to each priority based on how many degraded hosts are in that priority. This should be computed the same way as priority_load_ but looking at degraded hosts instead of healthy ones. These two should sum to a hundred the same way we're normalizing it today, with the healthy hosts being allocated load first. In the case of everything being unhealthy, we still route to P0 healthy.

With this, we can update chooseHostSet to iterate over both loads and return both the desired host set and whether we picked it from the degraded load. Using this information we can use this in hostSourceToUse to specify a that we want the degraded hosts from the subset.

In order to maintain locality routing, a separate locality weights value can be kept that adjusts the weights based on (degraded vs total hosts) in a locality. This would be used in hostSourceToUse to select a locality when locality weighting is enabled.

When in panic mode (be it per priority or global) degraded values should be ignored (route to all hosts) and degraded hosts should not contribute to the panic mode threshold. That is, if a priority is fully degraded it should not be in panic mode. This is because each host is still routable, so triggering panic mode isn't really useful.

[mattklein123]

@snowp this makes sense to me. I like the idea of reusing priorities for this as I think it makes the implementation a lot simpler and more intuitive given what we already have. Perhaps you could also tackle #5081 as part of this. 😉

[stale]

This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged "help wanted" or other activity occurs. Thank you for your contributions.

[snowp]

As part of this I'll have to update the health check filter to cache the response headers, or at least the degraded header. Currently the caching layer is causing degraded health checks to flap, making them incompatible.

[snowp]

This is now fully implemented.