htlcswitch: fix empty commit sig and no commit sig #2927

joostjager · 2019-04-10T19:25:34Z

Builds on top of #3564.

This PR fixes two known bugs in the commitment tx update sequence:

Sending an commit sig message that didn't sign any new updates ("empty commit sig")
Not sending a commit sig message when it is expected.

In addition to that, several simplifications are carried through where values were tracked in the link that can also be obtained from the channel.

halseth · 2019-04-15T11:43:26Z

For reference: #1927

@Roasbeef

The spec says not to send a commitment_signed without any changes, but LND does this. To understand why, you have to understand how LND works. I haven't read the code, but I'm pretty sure it works like this: 1. lnd slows down to do garbage collection, because it's in Go. 2. When an alert timer goes off, noticing it's not making process, it sends a twitter message to @Roasbeef. 3. @Roasbeef sshs into the user's machine and binary patches lnd to send a commitment_signed message. 4. Unfortunately he works so fast that various laws of causality are broken, meaning sometimes the commitment_signed is sent before any of thes other things happen. I'm fairly sure that this will stop as @Roasbeef ages, or lnd introduces some kind of causality enforcement fix. Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

joostjager · 2019-04-25T16:52:39Z

@halseth's repro of empty commitSig: #3012

cfromknecht · 2019-09-29T20:39:07Z

lnwallet/channel.go

merge with prior commit? should we also use this in l.debugf(), l.errorf(), etc? atm i think some use chan point and others use short channel id, would be nice if we were consistent

Hehe. Ok, I know that I sometimes comment on commit structure in PRs and suggest some of them to be merged. This is particular the case for structs that are introduced in a separate commit but it is hard to review the code without any further context. In this case the prefix logger is an object that doesn't need much context to review (although this is a sliding scale) and therefore I made it a separate commit.

Added commits to also make the link logging consistent. Open question is what we think is the best prefix to use for link and channel? uint64, shortchan, chanpoint?

cfromknecht · 2019-09-29T20:48:05Z

lnwallet/channel.go

code looks duplicated from earlier commit, perhaps make internal helper

But it isn't duplicated. Here we check if we owe a signature, while in ReceiveNewCommitment, we check whether the remote party owed anything.

build/prefix_log.go

htlcswitch/link.go

htlcswitch/link_isolated_test.go

halseth

Awesome change, making the channel state machine more deterministic and easy to reason about! LGTM 💯

htlcswitch/link.go

cfromknecht

Great work! Indeed this simplifies state machine and reduces the chance of bugs due to local state, e.g. needsUpdate 🍾

htlcswitch/link.go

lnwallet/channel.go

joostjager · 2019-10-29T20:04:16Z

PR is approved, but need to do interop testing with CL and Eclair

joostjager · 2019-10-31T15:37:41Z

Testing interop between CL 0.7.2 and this branch 5080a030968e783eb3e133ba5a3aa6c868169c95. Setup: connect CL and LND nodes with a channel. Then pay invoices both ways concurrently. Used script:

#!/bin/bash

while true
do
        LNCLI="$HOME/lightninglabs/lnd/lncli-debug --network=regtest"

        ID1=`od -vAn -N4 -tu4 < /dev/urandom`
        ID2=`od -vAn -N4 -tu4 < /dev/urandom`

        BOLT1=`lightning-cli invoice 10000 $ID1 x | jq -r .bolt11`
        BOLT2=`$LNCLI addinvoice 10 | jq -r .pay_req`

        $LNCLI --network=regtest payinvoice -f $BOLT1
        lightning-cli pay $BOLT2
done

I ran four instances of the script in parallel. No anomalies observed and channel remained functional.

joostjager · 2019-10-31T15:59:23Z

Same test for Eclair 0.3.2-5ad3944, no problems observed.

#!/bin/bash

while true
do
        LNCLI="$HOME/lightninglabs/lnd/lncli-debug --network=regtest"
        ECLAIRCLI="$HOME/Downloads/eclair-cli.sh -a localhost:8081 -p foobar"
        ID1=`od -vAn -N4 -tu4 < /dev/urandom`
        ID2=`od -vAn -N4 -tu4 < /dev/urandom`

        BOLT1=`$ECLAIRCLI createinvoice --amountMsat=10000 --description="test" | jq -r .serialized`
        BOLT2=`$LNCLI addinvoice 10 | jq -r .pay_req`

        $LNCLI --network=regtest payinvoice -f $BOLT1
        $ECLAIRCLI payinvoice --invoice=$BOLT2
done

joostjager · 2019-10-31T16:04:09Z

Finally interop with lnd 0.8. Works too.

joostjager · 2019-10-31T16:07:35Z

Moved PR back into the board's approved column. Ready for a final check

joostjager · 2019-11-05T15:55:17Z

Rebased

Roasbeef

Solid set of changes, no major comments on initial pass. I've let a comment on a older related PR, to see how we can reconcile the two of them. Only material comment is a shift to avoid grabbing the channel state machine's mutex twice when we receive a new commitment.

Roasbeef · 2019-11-06T04:15:56Z

lnwallet/channel.go

+// updates in the local commit tx that aren't reflected in the remote commit tx
+// yet.
+func (lc *LightningChannel) OweCommitment(local bool) bool {
+	lc.RLock()


Perhaps we should instead obtain the mutex once within ReceiveNewCommitment, then either make this a private method, or update the godoc to note that it expects the mutex to already be held?

It is also called without a lock from ChannelLink. I now created a private no-lock version and a public method that wraps it in a lock.

I am wondering whether our channel manipulations are always safe though. Something like this for example:

if lc.OweCommitment(true) { commitSig, htlcSigs, _, err := lc.SignNextCommitment()

I think the channel isn't modified concurrently, but why it the lock there then? Is it a safety precaution? Or for reading channel state in another subsystem?

The lock was mostly a safety precaution when the struct was first created just to ensure things were goroutine safe, so all other code added afterwards followed that trend. In the past there were also instances of grabbing live state machines from the peer, and using that to populate RPC information.

Roasbeef · 2019-11-06T04:26:00Z

htlcswitch/link.go

 			// pending updates we need to commit due to our
 			// commitment chains being desynchronized.
-			if l.channel.FullySynced() {
+			if !l.channel.OweCommitment(true) {


Ah, I see how this is used here now. Updating my prior comment, we can make a private version of this method that assumes the lock is already held, then call that when accepting a new commitment.

Roasbeef · 2019-11-06T04:42:13Z

htlcswitch/link.go

 			}

 		case <-l.cfg.BatchTicker.Ticks():
-			// If the current batch is empty, then we have no work


Nice, this should also lower idle CPU usage for routing nodes with tons of channels.

To facilitate the logging, this commit adds a new OweCommitment method. For the logging, we only need to consider the remote perspective. In a later commit, we'll also start using the local perspective to support the decision to send another signature.

Co-authored-by: Johan T. Halseth <johanth@gmail.com>

Previously the channel method FullySynced was used to decide whether to send a new commit sig message. However, it could happen that FullySynced was false, but that we didn't owe a commitment signature. Instead we were waiting on the other party to send us a signature. If that happened, we'd send out an empty commit sig. This commit modifies the condition that triggers a new commit sig and fixes this deviation from the spec.

Instead of tracking local updates in a separate link variable, query this state from the channel itself. This commit also fixes the issue where the commit tx was not updated anymore after a failed first attempt because the revocation window was closed. Also those pending updates will be taken into account when the remote party revokes.

Now that channel exposes the number of pending local updates, it is no longer necessary to track the batch size separately in the link.

Replace logCommitTick as a way to deal with revocation window exhaustion by retrying to update the commit tx when the remote revocation is received. The rationale is that the revocation window always opens up because of a revoke message that is received from the other party. It is therefore not necessary to set a timer for this. The reception of the revoke message is the trigger to send a new commit sig if necessary.

joostjager · 2019-11-06T08:24:15Z

Test from #3012 added to this PR. It needed a bit of modification, but the idea of the scenario is the same.

For review, this is the delta with the previous revision of this pr:
https://github.com/joostjager/lnd/compare/hodl-drop-fix-better-ref..joostjager:hodl-drop-fix-better

Roasbeef

LGTM 🎷

joostjager changed the title ~~htlcswitch: always send commitSig~~ htlcswitch: always send commitSig [wip] Apr 10, 2019

joostjager mentioned this pull request Apr 10, 2019

htlcswitch: ensure commit tx gets updated after settle hodl invoice #2918

Merged

cfromknecht added this to the 0.7 milestone Apr 11, 2019

joostjager mentioned this pull request Apr 11, 2019

htlcswitch: promote log level of revocation window exhaustion #2914

Closed

Roasbeef added bug fix htlcswitch commitments Commitment transactions containing the state of the channel labels Apr 20, 2019

cfromknecht removed this from the 0.7 milestone May 30, 2019

wpaulino added this to the 0.8.0 milestone Jul 25, 2019

wpaulino added the v0.8.0 label Jul 30, 2019

wpaulino removed the v0.8.0 label Sep 16, 2019

wpaulino removed this from the 0.8.0 milestone Sep 16, 2019

joostjager force-pushed the hodl-drop-fix-better branch from 2847230 to de5cd1d Compare September 24, 2019 12:36

joostjager requested review from Roasbeef, cfromknecht and halseth as code owners September 24, 2019 12:36

joostjager changed the title ~~htlcswitch: always send commitSig [wip]~~ htlcswitch: fix empty commit sig and no commit sig Sep 24, 2019

joostjager force-pushed the hodl-drop-fix-better branch from de5cd1d to f8a137c Compare September 24, 2019 12:39

joostjager removed the request for review from halseth September 24, 2019 12:40

joostjager force-pushed the hodl-drop-fix-better branch 2 times, most recently from 8b50fb3 to ab2078b Compare September 24, 2019 13:14

joostjager mentioned this pull request Sep 24, 2019

[WIP] link_test: empty sig testcase #3012

Closed

joostjager force-pushed the hodl-drop-fix-better branch 2 times, most recently from a073f13 to c230496 Compare September 26, 2019 13:11

cfromknecht reviewed Oct 1, 2019

View reviewed changes

joostjager force-pushed the hodl-drop-fix-better branch from c230496 to 6548569 Compare October 1, 2019 09:28

joostjager requested a review from cfromknecht October 1, 2019 09:39

joostjager force-pushed the hodl-drop-fix-better branch from d55cd10 to 831dfc2 Compare October 24, 2019 07:18

joostjager requested a review from halseth October 24, 2019 07:20

halseth approved these changes Oct 24, 2019

View reviewed changes

htlcswitch/link.go Outdated Show resolved Hide resolved

joostjager mentioned this pull request Oct 24, 2019

routing: non-strict path finding #3558

Merged

cfromknecht approved these changes Oct 29, 2019

View reviewed changes

htlcswitch/link.go Outdated Show resolved Hide resolved

lnwallet/channel.go Outdated Show resolved Hide resolved

joostjager force-pushed the hodl-drop-fix-better branch from 831dfc2 to 5080a03 Compare October 31, 2019 14:53

joostjager force-pushed the hodl-drop-fix-better branch from 5080a03 to 8a02365 Compare November 5, 2019 15:55

Roasbeef reviewed Nov 6, 2019

View reviewed changes

joostjager force-pushed the hodl-drop-fix-better branch 3 times, most recently from 7d856d1 to d0d6b20 Compare November 6, 2019 08:14

joostjager and others added 8 commits November 6, 2019 09:21

htlcswitch/test: add test cases that triggers empty commit sig

64f4421

Co-authored-by: Johan T. Halseth <johanth@gmail.com>

htlcswitch: stop batch timer if there are no updates

2482d84

htlcswitch: remove batch counter

5078d66

Now that channel exposes the number of pending local updates, it is no longer necessary to track the batch size separately in the link.

htlcswitch/test: test revocation window exhaustion

ae67b1a

joostjager force-pushed the hodl-drop-fix-better branch from d0d6b20 to 0b5afa6 Compare November 6, 2019 08:22

joostjager requested a review from Roasbeef November 6, 2019 08:24

Roasbeef approved these changes Nov 9, 2019

View reviewed changes

Roasbeef merged commit 86bed39 into lightningnetwork:master Nov 9, 2019

htlcswitch: fix empty commit sig and no commit sig #2927

htlcswitch: fix empty commit sig and no commit sig #2927

Uh oh!

Conversation

joostjager commented Apr 10, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

halseth commented Apr 15, 2019

Uh oh!

joostjager commented Apr 25, 2019

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

halseth left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

cfromknecht left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

joostjager commented Oct 29, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

joostjager commented Oct 31, 2019

Uh oh!

joostjager commented Oct 31, 2019

Uh oh!

joostjager commented Oct 31, 2019

Uh oh!

joostjager commented Oct 31, 2019

Uh oh!

joostjager commented Nov 5, 2019

Uh oh!

Roasbeef left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

joostjager commented Nov 6, 2019

Uh oh!

Roasbeef left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

joostjager commented Apr 10, 2019 •

edited

Loading

joostjager commented Oct 29, 2019 •

edited

Loading