Dtab alternate not chosen over higher-precedence failing service

[ncabatoff]

Expected behavior

Suppose I have a dtab

  /svc => /$/namer/srv1 | /$/namer/srv2

I would expect that if every request issued to srv1 resulted in a failure, requests would start being sent to srv2 instead.

Actual behavior

So long as srv1 can be bound to an address, it receives all the traffic.

Is there any way to make it so that srv1 failures will result in requests being routed to srv2? I want srv1 to get traffic preferentially, and I don't want to rely solely on a namer's active health checks to control that.

[elecnix]

Commit 808a935 shows a prototype implementing this behavior.

[nepthar]

This is an interesting concept. As you point out, our dtab alt behavior is to only use the alt when resolution fails, as opposed to when service creation fails. This is (as far as I can tell) intentional and separate from the way we typically handle these kinds of resiliency things (see circuit breaking, for instance).

@elecnix have you tested/run your proof of concept?

If you folks decide to build out this feature, I think it's important to very carefully think through:

• What constitutes a "failure"? Loss of connectivity to the primary serverset?
• Once in a failure scenario, how do we return to a healthy state?

Finally, one more thought - if this sort of functionality is something you're looking to get up and running ASAP, it could be implemented by having two clients - one for the primary and one for the secondary. You could track successes and switch between clusters as necessary.

@roanta I'd love to hear your thoughts on this, too.

[elecnix]

The change was validated in a simple integration test.

This implementation relies on the existing circuit breaking. When the circuit "opens", the service factory "closes", and the Alted service factory picks the next available circuit. Once the circuit breaker starts probing, the Alted factory sends an equivalent request.

This is inherently racy. I'm okay with that, but open to suggestions.

As a disclaimer, I'm running this change with an unmodified Linkerd. I'll think about the idea of creating two clients, but I doubt that would fit nicely with how they integrate Finagle.