[reliable payments] router payment state machine by halseth · Pull Request #2761 · lightningnetwork/lnd

halseth · 2019-03-12T13:25:29Z

This PR introduces a persistent state machine to the ChannelRouter's payment flow, ensuring we can handle payment results that are received after a restart. It also paves they way for adding cancellation of payments, and resuming payment sessions across restarts.

Problem

lnd currently runs into problems if it is restarted while an HTLC is in flight on the network. The primary reason for this is the way the router hands the HTLC to the switch. The router persist no information about the payment, so when a result eventually comes back, we risk the information needed to properly populate the OutgoingPayment in the database is lost.

Solution

With the paymentStateMachine introduced in this PR, we store two pieces of key information:

The HTLC paymentID. This is used to query the Switch whether the HTLC is still active. If not active we know that we can safely retry the payment attempt. If active the router will wait for the result to be available, and store the result to the DB.
The HTLC's route. This is added to the DB together with the preimage when the payment succeeds.

Note

The Switch does not currently persist the pending payment attempts across restarts. This will be added in a follow-up PR.

Replaces #2475

Builds on

Roasbeef

I really dig this approach! With the state machine, I find it much easier to follow than the prior attempts at a solution to this problem. I've completed an initial pass so far, and the main question in my mind is the size of the overlap between this new state machine and the existing control tower in the switch. At one point the control tower addressed a need within the codebase, but it seems like this new state machine can eventually subsume the responsibilities of the control tower.

joostjager

I am, as always, worried about the persistence and the work and inflexibility it may cause us in the future. Tried to mainly analyze this part of the PR.

Why don't you merge the switch pr first btw, work bottom up? Or will we have with just this PR already good working resumes of payments?

joostjager · 2019-03-20T10:12:12Z

Looks unused

joostjager

Definitely nicer without that intermediate persistent state.

Main comments:

Code structure in router
Consolidation of payment related stores

cfromknecht

nice refactor of the router/switch interactions, this should make the whole payment flow much tighter!

one question i have with new state introduced in the control tower, will we just treat all payments that are currently grounded as started but never attempted? an alternative would be to rename grounded payments as failed, and introduce a new state for initiated. not sure which is better atm

cfromknecht · 2019-03-21T20:08:28Z

+		return nil
+	}
+
+	ctx.router.cfg.GetPaymentResult = func(paymentID uint64) (


many of these GetPaymentResult funcs seem similar, any way we can generate the closures w/ less code duplication?

This commit changes the format used to store payments within the DB. Previously this was serialized as one continuous struct OutgoingPayment, which also contained an Invoice struct we where only using a few fields of. We now split it up into two simpler sub-structs CreationInfo, AttemptInfo and PaymentPreimage. We also want to associate the payments more closely with payment statuses, so we move to this hierarchy: There's one top-level bucket "sentPaymentsBucket" which contains a set of sub-buckets indexed by a payment's payment hash. Each such sub-bucket contains several fields: paymentStatusKey -> the payment's status paymentCreationInfoKey -> the payment's CreationInfo. paymentAttemptInfoKey -> the payment's AttemptInfo. paymentSettleInfoKey -> the payment's preimage (or zeroes for non-settled payments) The CreationInfo is information that is static during the whole payment lifcycle. The attempt info is set each time a new payment attempt (route+paymentID) is sent on the network. The preimage is information only known when a payment succeeds. It therefore makes sense to split them. We keep legacy serialization code for migration puproses.

migrateOutgoingPayments moves the OutgoingPayments into a new bucket format where they all reside in a top-level bucket indexed by the payment hash. In this sub-bucket we store information relevant to this payment, such as the payment status. To avoid that the router resend payments that have the status InFlight (we cannot resume these payments for pre-migration payments) we delete those statuses, so only Completed payments remain in the new bucket structure.

Since we have performed a migration, the db should be in a consistent state, and we can remove the non-strict option.

This commit gives a new responsibility to the control tower, letting it populate the payment bucket structure as the payment goes through its different stages. The payment will transition states Grounded->InFlight->Success/Failed, where the CreationInfo/AttemptInfo/Preimage must be set accordingly. This will be the main driver for the router state machine.

This encapsulates all state needed to resume a payment from any point of the payment flow, and that must be shared between the different stages of the execution. This is done to prepare for breaking the send loop into smaller parts, and being able to resume the payment from any point from persistent state.

This commit makes the router use the ControlTower to drive the payment life cycle state machine, to keep track of active payments across restarts. This lets the router resume payments on startup, such that their final results can be handled and stored when ready.

On startup the router will fetch the in-flight payments from the control tower, and resume their execution.

TestRouterPaymentStateMachine tests that the router interacts as expected with the ControlTower during a payment lifecycle, such that it payment attempts are not sent twice to the switch, and results are handled after a restart.

And unexport deprecated code.

TestPaymentControlDeleteNonInFlight checks that calling DeletaPayments only deletes payments from the database that are not in-flight.

halseth added this to the 0.6 milestone Mar 12, 2019

halseth added the payments Related to invoices/payments label Mar 12, 2019

This was referenced Mar 12, 2019

[WIP] Router payments persistence #2475

Closed

[reliable payments] persist htlcswitch pending payments #2762

Merged

halseth force-pushed the reliable-payments-router-state-machine branch from e83fd93 to 2409e3d Compare March 12, 2019 13:55

Roasbeef requested changes Mar 15, 2019

View reviewed changes

halseth mentioned this pull request Mar 19, 2019

lnrpc+rpcserver: display payreq via listpayments #2717

Merged

halseth force-pushed the reliable-payments-router-state-machine branch 7 times, most recently from 78a0c81 to 1687cdc Compare March 20, 2019 10:22

joostjager reviewed Mar 20, 2019

View reviewed changes

halseth force-pushed the reliable-payments-router-state-machine branch 5 times, most recently from ce35b99 to 9d1bd42 Compare March 21, 2019 14:58

joostjager reviewed Mar 21, 2019

View reviewed changes

Comment thread channeldb/payments.go Outdated

Comment thread routing/control_tower.go Outdated

Comment thread routing/payment_store.go Outdated

Comment thread routing/router.go Outdated

Comment thread routing/router.go Outdated

Comment thread channeldb/payments.go Outdated

cfromknecht reviewed Mar 22, 2019

View reviewed changes

halseth force-pushed the reliable-payments-router-state-machine branch 3 times, most recently from 7180772 to 7aae12a Compare March 26, 2019 10:11

joostjager reviewed Mar 26, 2019

View reviewed changes

Comment thread channeldb/payments.go Outdated

Comment thread routing/router.go Outdated

joostjager reviewed Mar 26, 2019

View reviewed changes

Comment thread channeldb/control_tower.go Outdated

halseth force-pushed the reliable-payments-router-state-machine branch 2 times, most recently from fd20644 to ab0f77c Compare March 26, 2019 14:38

Roasbeef removed this from the 0.6 milestone Mar 26, 2019

halseth added 27 commits May 27, 2019 20:18

channeldb/codec: add privateKey serialization

f022810

channeldb/control_tower: add FetchSentPayments

693807c

channeldb test: add TestOutgoingPaymentsMigration

b7189ba

channeldb/control_tower: remove non-strict option

bb4aadd

Since we have performed a migration, the db should be in a consistent state, and we can remove the non-strict option.

channeldb/payments: add StatusFailed

6d80661

channeldb/control_tower test: test new payment behavior

59c2557

routing/router: move sending and receiving payment result into loop

ae7bf2c

routing/payment_lifecycle: extract error handling into method

e9b2182

routing/payment_lifecycle: extract create attempt into method

dd73c51

routing/control_tower: add FetchInFlightPayments

677cb01

routing/router: resume payment state machine at startup

3323800

On startup the router will fetch the in-flight payments from the control tower, and resume their execution.

routing: export MissionControl

adc4640

routing/payment_session+router: make PaymentSession interface

f4306b1

routing+server: define PaymentSessionSource

3f76bc0

routing/router_test: add TestRouterPaymentStateMachine

60e2367

TestRouterPaymentStateMachine tests that the router interacts as expected with the ControlTower during a payment lifecycle, such that it payment attempts are not sent twice to the switch, and results are handled after a restart.

channeldb+router: record payment failure reason in db

1b78890

channeldb: move deprecated code to migration_09_legacy_serialization.go

d78d3f5

And unexport deprecated code.

channeldb+multi: rename Grounded->Unknown, Completed->Succeeded

d705b8a

channeldb: derive PaymentStatus implicitly from stored info

95b4828

control_tower: make initial bucket bucket creation stricter

589f0fc

routing/router_test: add test case for ForwardingError on SendHTLC

649ee87

channeldb/control_tower test: add TestPaymentControlDeleteNonInFlight

7cb25a5

TestPaymentControlDeleteNonInFlight checks that calling DeletaPayments only deletes payments from the database that are not in-flight.

treygriffith mentioned this pull request Jun 8, 2019

Resolved hashes in payments do not reflect back their settled state #2032

Closed

Roasbeef mentioned this pull request Jul 2, 2019

[WIP] channeldb+routing+switch: move payment tracking from rpcserver to router+switch #2246

Closed

halseth mentioned this pull request Aug 21, 2020

Async payment cancellation #4562

Open

Uh oh!

Conversation

halseth commented Mar 12, 2019 • edited by cfromknecht Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem

Solution

Note

Uh oh!

Roasbeef left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

joostjager left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

joostjager Mar 20, 2019

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

joostjager left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cfromknecht left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

cfromknecht Mar 21, 2019

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

halseth commented Mar 12, 2019 •

edited by cfromknecht

Loading

cfromknecht left a comment •

edited

Loading