[reliable payments] persist htlcswitch pending payments #2762

halseth · 2019-03-12T13:45:32Z

This PR is a follow-up from #2761. It adds a persistent pendingPaymentStore to the Switch, needed to store payment results when the router is not available to handle them.

Problem

After a restart, there is no connection between an HTLC in flight on the network and the router payment flow that initially sent the HTLC. The router will eventually call GetPaymentResult to retrieve the result of the HTLC, but we have no guarantees this will be done before the result is received. This can lead to the result getting dropped, which is not ideal.

Solution

We introduce a pendingPayementStore which has two primary functions:

It stores PaymentResults when they are received, regardless of whether there are any subscribers (the ChannelRouter) to the corresponding HTLC. This ensures that the router gets handed the result when it calls GetPaymentResult.
It stores the information necessary to identify and handle a result that comes back after a restart. This is the chanID and paymentID, used to identify the payment in the circuit map.
On restarts it syncs the pending payments with the CircuitMap. We use the circuit map as the source of truth whether a payment is in flight on the network. Syncing it at startup, before we start handling HTLCs, is necessary to be able to decide whether an HTLC was successfully forwarded.

Builds on #2761
Replaces #2265

channeldb/payments.go

Roasbeef

I find this new approach much easier to follow than the prior iterations! I'm not sure we can get rid of the control tower as is though, due to the possibility of a user initiating a payment, then updating to this new version. Worst case they would re-try with the same payment hash and possibly lose that payment amount.

First pass review completed, will need some additional unit tests for the new biz logic in the pending payment store, and also the new functionality w.r.t passing back the same payment ID, etc.

htlcswitch/switch.go

htlcswitch/pending_payment.go

htlcswitch/switch.go

htlcswitch/pending_payment.go

Roasbeef · 2019-03-20T03:08:46Z

htlcswitch/switch.go

+	// network. By deleting those pending payments that are not found in
+	// the circuit map, we ensure that the router won't wait for a result
+	// to be returned for this payment ID.
+	// NOTE: this assumes that the circuit map has already been trimmed,


I don't think this last assumption is sound with the current implementation. IIRC, the circuit map as is, can contain circuits for payments they have either already been completed or failed on the network.

cc @cfromknecht

joostjager

First pass review

joostjager · 2019-03-20T10:26:49Z

htlcswitch/pending_payment.go

+// fetchPendingPayments retrieves all payments from the store that are pending,
+// meaning no results are available yet, and can be re-forwarded on the
+// network.
+func (store *pendingPaymentStore) fetchPendingPayments() ([]*PendingPayment,


Looks like this is test only code. Would look for a different way to assert behaviour (intercepting mock?)

joostjager · 2019-03-20T10:29:16Z

htlcswitch/pending_payment.go

+
+// getPayment is used to query the store for a pending payment for the given
+// payment ID. If the pid is not found, ErrPaymentIDNotFound will be returned.
+func (store *pendingPaymentStore) getPayment(pid uint64) (*PendingPayment,


Is this function used?

joostjager · 2019-03-20T10:34:40Z

htlcswitch/pending_payment.go

+// syncWithCircuitMap should be called at startup to delete the pending
+// payments which are no longer found in the circuit map. This removes pending
+// payments that was not successfully forwarded to the link before shutdown.
+func (store *pendingPaymentStore) syncWithCircuitMap(


If we need to do this anyway, what is the use of persisting pending payments? Can't that list be reconstructed on startup from the circuit map?

So only persist the outcomes when they become available.

joostjager · 2019-03-20T10:37:47Z

htlcswitch/pending_payment.go

+
+// completePayment stores the PaymentResult for the given paymentID, and
+// notifies any subscribers.
+func (store *pendingPaymentStore) completePayment(paymentID uint64,


Are those results left in forever?

joostjager · 2019-05-06T08:54:45Z

htlcswitch/switch.go

+	// If the provided deobfuscator is nil, we have discarded the error
+	// decryptor due to a restart. We'll return a fixed error and signal a
+	// temporary channel failure to the router.
+	case deobfuscator == nil:
 		userErr := fmt.Sprintf("error decryptor for payment " +
 			"could not be located, likely due to restart")


How is this still possible after this PR?

joostjager · 2019-05-06T11:51:43Z

htlcswitch/payment_result.go

@@ -0,0 +1,268 @@
+package htlcswitch


Rename file to payment_result_store.go, as we do with other stores too?

joostjager · 2019-05-06T11:53:44Z

htlcswitch/payment_result.go

+
+	// Otherwise, write 'true' and the error fields.
+	msg := []byte(p.Error.ExtraMsg)
+	err := channeldb.WriteElements(w, true,


Saving the human readable error ExtraMsg to the db is not ideal. There are only a few cases for ExtraMsg, those can probably be saved in a structured way.

Upon further inspection, there may be a few more. But still doubt whether we should persist data just to get a nice error message. Maybe just logging when the error occurs is enough.

Yeah, looks like they are only used locally for passing the case of error to the RPC. Would it be useful to define a field for local error codes that can be used to communicate local failures?

I think that local internal errors (things that should never happen) don't need to be exposed over rpc. Logging and returning a generic internal error should be enough?

joostjager · 2019-05-06T11:55:35Z

htlcswitch/payment_result.go

+	// Otherwise, write 'true' and the error fields.
+	msg := []byte(p.Error.ExtraMsg)
+	err := channeldb.WriteElements(w, true,
+		p.Error.ErrorSource, msg,


ErrorSource could be nil if the error source is unknown (for example after restart or when the onion decrypt fails). The store should probably prepare for that, to prevent a db migration.

Using nil can leads to problems down the line, what do you think about defining a constant UnknownPaymentSource?

What kind of problems? nil for unset fields is quite common? For serialize to the db we can of course write anything we want.

htlcswitch/payment_result.go

htlcswitch/switch.go

halseth · 2019-05-28T12:51:40Z

Rebased.

Roasbeef · 2019-05-29T00:51:29Z

First commit can be removed during the next rebase.

htlcswitch/payment_result.go

Roasbeef · 2019-05-29T01:10:05Z

htlcswitch/payment_result.go

+	var paymentIDBytes [8]byte
+	binary.BigEndian.PutUint64(paymentIDBytes[:], paymentID)
+
+	err := store.db.Batch(func(tx *bbolt.Tx) error {


May want to consider making this an update instead, if only for the sake of the integration tests. In the past, then extra 30ms or so jitter here really accumulated making many tests time out when we tried Batching everywhere before.

Yeah, I'm def in favor of using Update as long as we don't have any clear indication that it is a performance bottleneck.

if we do decide to use Update, then we probably don't need a multimutex since the db txns can't execute concurrently. storeResult is called in separate goroutines, so under heavy load i suspect Batch would help, but would need to benchmark to be certain

Still using a Update here, but haven't had a chance to attempt to ascertain the performance impact. I think we can just leave it as it is, then during the rc process possibly adjust it if we find that it makes a significant impact.

htlcswitch/switch.go

cfromknecht · 2019-05-29T23:53:43Z

htlcswitch/switch.go

@@ -989,6 +976,7 @@ func (s *Switch) parseFailedPayment(deobfuscator ErrorDecrypter,
 	// the first hop. In this case, we'll report a permanent
 	// channel failure as this means us, or the remote party had to
 	// go on chain.
+	// TODO: check reason length instead


what is meant by this/when do we plan to do this?

cfromknecht · 2019-05-29T23:55:48Z

htlcswitch/switch_test.go

+		t.Fatalf("unable to get payment result: %v", err)
+	}
+
+	select {


not blocking, but this segment is an exact copy of the logic above, maybe add a checkResult closure?

cfromknecht · 2019-05-29T23:57:27Z

Really happy with how this refactor turned out, this method of storing the payment results i think is much cleaner than some of the avenues we explored earlier! Will need rebase now that #3087 has landed

cfromknecht · 2019-05-30T00:02:52Z

lnd_test.go

+	}
+
+	// Wait for all the invoices to reach the OPEN state.
+	for _, stream := range invoiceStreams {


at some point these helper routines for hodl invoices could be extracted, but not blocker on this PR

joostjager · 2019-06-04T11:46:38Z

htlcswitch/switch.go

+		msg:          pkt.htlc,
+		unencrypted:  unencrypted,
+		isResolution: pkt.isResolution,
+	}


Add timestamp to record when failure was received. This is one building block of black hole defense

Preferably time.UnixNano for sub-second resolution

AFAIK, we don't store time stamps when the payment is sent, so what good will it to do store them here? Even then, I think we can store all the timestamps at the router level rather than the switch since although they're able to operate independently with the new design, they operate in near unison typically.

In any case, rather than piggy backing on the PR, I think this can be done as a distinct change once we start to center in on a more stable design.

Roasbeef · 2019-06-06T11:55:41Z

Can now be updated to use the latest RPC calls in the new sub-server for the new itests.

Roasbeef

LGTM ☄️

Two non-blocking follow up that I'd like to see extended as a PR soon after this lands:

Additional itest for the payment failure case
Clean up of the payment result once that router updates its state, and ACKs the message from the switch.

Roasbeef · 2019-06-07T11:40:27Z

htlcswitch/payment_result.go

+	var paymentIDBytes [8]byte
+	binary.BigEndian.PutUint64(paymentIDBytes[:], paymentID)
+
+	err := store.db.Batch(func(tx *bbolt.Tx) error {


Still using a Update here, but haven't had a chance to attempt to ascertain the performance impact. I think we can just leave it as it is, then during the rc process possibly adjust it if we find that it makes a significant impact.

Roasbeef · 2019-06-07T11:47:26Z

htlcswitch/switch.go

+		msg:          pkt.htlc,
+		unencrypted:  unencrypted,
+		isResolution: pkt.isResolution,
+	}


AFAIK, we don't store time stamps when the payment is sent, so what good will it to do store them here? Even then, I think we can store all the timestamps at the router level rather than the switch since although they're able to operate independently with the new design, they operate in near unison typically.

In any case, rather than piggy backing on the PR, I think this can be done as a distinct change once we start to center in on a more stable design.

cfromknecht

LGTM! 🔥

paymentResultStore is a persistent store where we keep track of all received payment results. This is used to ensure we don't lose results from payment attempts on restarts.

Used for logging in the switch, and when we remove the pending payments, only the router will have the hash stored across restarts.

testHoldInvoicePersistence tests that a sender to a hold-invoice, can be restarted before the payment gets settled, and still be able to receive the preimage.

TestNetworkResultStore tests that the networkResult store behaves as expected, and that we can store, get and subscribe to results.

TestSwitchGetPaymentResult tests that the switch interacts as expected with the circuit map and network result store when looking up the result of a payment ID. This is important for not to lose results under concurrent lookup and receiving results.

halseth · 2019-06-07T14:55:14Z

Alright, itest using the new payment API is pushed. PTAL @joostjager

halseth added this to the 0.6 milestone Mar 12, 2019

halseth force-pushed the reliable-payments-lookup-circuitmap branch 6 times, most recently from c22bf88 to 298a129 Compare March 12, 2019 14:39

alexbosworth reviewed Mar 12, 2019

View reviewed changes

channeldb/payments.go Outdated Show resolved Hide resolved

Roasbeef requested changes Mar 20, 2019

View reviewed changes

joostjager reviewed Mar 20, 2019

View reviewed changes

halseth force-pushed the reliable-payments-lookup-circuitmap branch from 298a129 to cb790c4 Compare March 20, 2019 13:26

This was referenced Mar 21, 2019

bug: invoice is not settled, but funds were received over on-chain htlc sweep #2811

Closed

htlcswitch: reliable HTLC tracking #2265

Closed

Roasbeef removed this from the 0.6 milestone Mar 26, 2019

joostjager mentioned this pull request Apr 1, 2019

Loop In lightninglabs/loop#34

Merged

halseth force-pushed the reliable-payments-lookup-circuitmap branch 3 times, most recently from aea2217 to 858ffd8 Compare April 9, 2019 09:30

cfromknecht added this to the 0.7 milestone Apr 11, 2019

joostjager reviewed May 6, 2019

View reviewed changes

halseth force-pushed the reliable-payments-lookup-circuitmap branch 2 times, most recently from 39263d4 to e24ded1 Compare May 6, 2019 10:19

joostjager reviewed May 6, 2019

View reviewed changes

halseth force-pushed the reliable-payments-lookup-circuitmap branch 2 times, most recently from 7c2f8d7 to 8294a42 Compare May 6, 2019 12:31

joostjager reviewed May 7, 2019

View reviewed changes

htlcswitch/payment_result.go Show resolved Hide resolved

htlcswitch/switch.go Outdated Show resolved Hide resolved

htlcswitch/switch.go Outdated Show resolved Hide resolved

halseth force-pushed the reliable-payments-lookup-circuitmap branch 4 times, most recently from 0975df7 to 28401bb Compare May 9, 2019 11:49

halseth force-pushed the reliable-payments-lookup-circuitmap branch from 7195371 to e9c25ae Compare May 28, 2019 11:56

halseth mentioned this pull request May 28, 2019

[reliable payments] switch result store clean-up #3131

Merged

Roasbeef requested changes May 29, 2019

View reviewed changes

halseth force-pushed the reliable-payments-lookup-circuitmap branch from e9c25ae to bc77400 Compare May 29, 2019 10:27

joostjager mentioned this pull request May 29, 2019

routerrpc: async payment rpc #2973

Merged

Roasbeef mentioned this pull request May 29, 2019

Settling a hold invoice while the sender is offline results in the sender not persisting the preimage #3135

Closed

cfromknecht reviewed May 29, 2019

View reviewed changes

cfromknecht reviewed May 30, 2019

View reviewed changes

joostjager reviewed Jun 4, 2019

View reviewed changes

halseth force-pushed the reliable-payments-lookup-circuitmap branch from bc77400 to 88baef2 Compare June 6, 2019 15:01

Roasbeef approved these changes Jun 7, 2019

View reviewed changes

cfromknecht approved these changes Jun 7, 2019

View reviewed changes

htlcswitch/mock: set SelfKey and mock ErrorSource

f5dee02

halseth force-pushed the reliable-payments-lookup-circuitmap branch from b4ea245 to 78dfb83 Compare June 7, 2019 14:42

halseth added 7 commits June 7, 2019 16:53

htlcswitch/payment_result: add (de)serialization of networkResult + test

df3f5d0

htlcswitch/payment_result: add paymentResultStore

1febe1a

paymentResultStore is a persistent store where we keep track of all received payment results. This is used to ensure we don't lose results from payment attempts on restarts.

multi: make GetPaymentResult take payment hash

2dea790

Used for logging in the switch, and when we remove the pending payments, only the router will have the hash stored across restarts.

htlcswitch/switch: use paymentResultStore to keep track of results

2cc778d

lnd_test: add testHoldInvoicePersistence

f556b37

testHoldInvoicePersistence tests that a sender to a hold-invoice, can be restarted before the payment gets settled, and still be able to receive the preimage.

htlcswitch/payment_result_test: add TestNetworkResultStore

dd3abbc

TestNetworkResultStore tests that the networkResult store behaves as expected, and that we can store, get and subscribe to results.

halseth force-pushed the reliable-payments-lookup-circuitmap branch from 78dfb83 to dd88015 Compare June 7, 2019 14:53

halseth merged commit e45d4d7 into lightningnetwork:master Jun 8, 2019

treygriffith mentioned this pull request Jun 8, 2019

Resolved hashes in payments do not reflect back their settled state #2032

Closed

[reliable payments] persist htlcswitch pending payments #2762

[reliable payments] persist htlcswitch pending payments #2762

Conversation

halseth commented Mar 12, 2019 • edited Loading

Problem

Solution

Roasbeef left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

joostjager left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

halseth commented May 28, 2019

Roasbeef commented May 29, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cfromknecht commented May 29, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Roasbeef commented Jun 6, 2019 • edited Loading

Roasbeef left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cfromknecht left a comment

Choose a reason for hiding this comment

halseth commented Jun 7, 2019

halseth commented Mar 12, 2019 •

edited

Loading

Roasbeef commented Jun 6, 2019 •

edited

Loading