routing: failure attribution

[joostjager]

This PR overhauls the interpretation of failed payments. It changes the interpretation rules so that we always apply the strongest possible set of penalties, without making assumptions that would hurt good nodes.

Main changes are:

• Apply different rule sets for intermediate and final nodes. Both types of nodes have different sets of failures that we expect. Penalize nodes that send unexpected failure messages.

• Distinguish between direct payments and multi-hop payments. For direct payments, we can infer more about the performance of our peer because we trust ourselves.

• In many cases it is impossible for the sender to determine which of the two nodes in a pair is responsible for the failure. In this situation, we now penalize bidirectionally. This does not hurt the good node of the pair, because only its connection to a bad node is penalized.

• Previously we always penalized the outgoing connection of the reporting node. This is incorrect for policy related failures. For policy related failures, it could also be that the reporting node received a wrongly crafted htlc from its predecessor. By penalizing the incoming channel, we surely hit the responsible node.

• FailExpiryTooSoon is a failure that could have been caused by any node up to the reporting node by delaying forwarding of the htlc. We don't know which node is responsible, therefore we now penalize all node pairs in the route. By doing this, we do hurt good nodes. But unfortunately there is currently no way to pinpoint just a single pair to penalize.

[joostjager]

note to self: handle private channels

[joostjager]

private_channels itest now failing, going to look into that.

[joostjager]

private_channels was another interesting failing test that surfaced that we only query bandwidth hints once per payment. In this test, two concurrent payments are executed. Because we stop tracking our own reputation in this pr, not updating the bandwidth hints led to an endless loop. Added a commit to update bandwidth hints with every attempt.

[cfromknecht]

Because we stop tracking our own reputation in this pr, not updating the bandwidth hints led to an endless loop.

Is this infinite loop still possible if an outdated node sends back tempchanfailure instead of invalidonionkey?

[Roasbeef]

In this test, two concurrent payments are executed. Because we stop tracking our own reputation in this pr,

Doesn't this series of bugs introduced show that maybe we shouldn't ignore our own channels at the router level? These isolated cases in our existing integration tests have been resolved, but whose to say there aren't other regressions awaiting?

[Roasbeef]

Because we stop tracking our own reputation in this pr, not updating the bandwidth hints led to an endless loop.

Why does it loop endlessly rather than just fail? (tentative LGTM...but want to know why this happened)

[joostjager]

Is this infinite loop still possible if an outdated node sends back tempchanfailure instead of invalidonionkey?

No, because the infinite loop could only happen when we got a tempchanfail from the self node. The self node cannot be outdated. For all other nodes on the route, we apply penalties.

Doesn't this series of bugs introduced show that maybe we shouldn't ignore our own channels at the router level? These isolated cases in our existing integration tests have been resolved, but whose to say there aren't other regressions awaiting?

Imo it shows that bugs have been able to go undetected because we penalize our own channels to create a vaguely defined safety net. Fixing those bugs doesn't just serve the changes in this pr, but makes the performance of lnd really better. For example, querying bandwidth hints only once per payment may have been causing problems with payments sometimes failing in a setting with concurrent payments.

I am still convinced that we should move forward with not penalizing ourselves. Imo there isn't a fundamental reason why would need to do that and it does create a complex interaction between switch, (global) mission control and concurrent payments that is hard to understand.

I do agree that other latent bugs may surface because of this change. In that case we will log a warning, but it could be that an endless loop occurs. Not the worst thing that can happen and with a one minute payment loop timeout it doesn't last that long, but it isn't pretty.

We could add some kind of circuit breaker that for example terminates the payment loop if more than three times the same route is attempted and log an error. But it needs to be kept in mind that three consecutive failures with identical routes could also happen without any bugs: If every time a channel has balance during path finding, but not anymore during execution (because of concurrent payments) AND a payment comes in thereafter to refill the channel before the next path finding round.

Also, the circuit breaker itself may again hide bugs because users ignore the error in the log. And it may never trip because there are no more bugs.

I tend towards reviewing the local dispatch execution path once more for any missed self-failure cases and then merging this without protection. What are your thoughts?

[joostjager]

Why does it loop endlessly rather than just fail? (tentative LGTM...but want to know why this happened)

It queried the bandwidth hints once at the start of the payment process. Bandwidth was sufficient at that point. Then a concurrent payment took the balance. The first payment process then started its attempt loop. An endless loop occurred because it tried the same depleted channel over and over again without updating the bandwidth hints. If it would have done so, it would have known that the channel had been emptied and would have given up.

[joostjager]

Traced the local dispatch/response path and checked all locally generated onion failure messages in switch.handleLocalDispatch, link.handleDownStreamPkt, link.handleUpstreamMsg and switch.GetPaymentResult (and the functions they call). Error cases can be divided in three groups:

1. Errors that fail the link -> will be picked up in next path finding round
2. Errors that are checked for during path finding too (for example max_htlc) -> will be picked up in next path finding round
3. Unexpected errors that should not happen. Examples:
   • We can't deserialize our own locally serialized failure
   • channel.AddHtlc fails (no overflow)
   • Can not get our own channel policy
     -> may trigger retry or an endless loop
4. Errors that could happen, but aren't picked up during path finding -> I found none of those

Assuming my assessment of this is correct, a potential endless loop can only occur for errors that shouldn't happen (3). I think that is acceptable. An endless loop (until timeout) isn't the worst that can happen and in those cases, there may be instability on other areas of lnd as well.

Further into the future, we may want to reevaluate how we handle those unexpected errors. Better define how fail-over behavior should be implemented, for example by failing the link instead of just logging an error. Or alternatively don't support fail-over at all and exit lnd completely in case of an undefined state.

[Roasbeef]

Thanks for investigating this in detail @joostjager! In the wild, it seems that even if we encounter something within the 3rd category, it won't loop endlessly as we still have the payment timeout parameter. In any case, as you pointed out, re-evaluating the bandwidth hints each time should address any issues that can arise today with multiple concurrent payments, which'll really help multi-user lnd instances such as some of the popular custodial wallets out there.

[Roasbeef]

LGTM 👽

[subhramazumdar]

Hi @joostjager. I was going through the penalization mechanism of failed payments. I don't find any mention of the same BOLT (sorry may be I have missed it). I see this has already been pushed in the lnd codebase so is it somehow working inherently? Can you elaborate on how it works? How are you deciding the penalty value?

[joostjager]

There is no BOLT for the penalization mechanism. It is up to the implementation team to decide.

[subhramazumdar]

@joostjager do you have any documentation on how this would work, if implemented? And how does penalty value gets decided for handling each category of failure?

[joostjager]

There is just the code and the comments in the code. You'll have to figure it out from there.

[joostjager]

Another resource is my presentation at lnconf 2019 in Berlin, https://twitter.com/joostjgr/status/1186177262238031872

[subhramazumdar]

Thanks for sharing.