routing: cancelable payment loop #8734

hieblmi · 2024-05-07T12:07:34Z

While interrupting lncli estimateroutefee ends the client program, the route server is left behind with an abandoned payment loop.

To allow users to cancel payment requests properly, the payment loop in resumePayment has to be abandoned as well as soon as the user request it.

lnd/routing/payment_lifecycle.go

Line 170 in f523f52

func (p *paymentLifecycle) resumePayment() ([32]byte, *route.Route, error) {

This PR makes the SendPaymentV2 streaming context accessible from the payment loop and allows it to safely end if the context was cancelled.

routing/payment_lifecycle.go

lnrpc/routerrpc/router_server.go

ellemouton

Looking good!

Only one blocking comment around removing the context.Context nil check.

The other comment is just a suggestion

routing/payment_lifecycle.go

ellemouton

LGTM! 🚀

yyforyongyu

Don't we have timeout already? Why do we need this? I think once the payment is sent, there's no easy way to cancel it, maybe related #5461

hieblmi · 2024-05-11T09:22:00Z

That's also my understanding, we can't cancel the payment once the htlc is sent. Here, additionally to the existent timeout check that interrupts the payment loop on expiry, we pass in a context (from the rpc server client stream) that is checked for cancellation and interrupts the payment loop if the user cancelled the stream context(Ctrl+C'd the sendpayment from cli). This can only happen if a p.sendAttempt(attempt) completed.

bitromortac

Code wise this looks good 🔥! Wanted to note that this code may break usage patterns, if users expect to be able to dispatch payments in the background with lncli sendpayment or somehow if SendPaymentV2 is not waited on for responses. This would not be possible anymore with the current setting as it stops the payment loop in the next occasion if the context is cancelled. If the cmd is interrupted, this will lead to cancellation of the payment loop, as expected. Should we retain backward compatibility by adding a bool cancellable to the payment request (we could then decide whether we'd like to forward the context or to initialize a new background one)?

routing/payment_lifecycle.go

routing/payment_lifecycle_test.go

routing/router.go

bhandras · 2024-05-14T13:12:16Z

routing/payment_lifecycle.go

+			return err
+		}
+
+	case <-ctx.Done():


I'd argue, that if the context times out we could instead fail the payment with FailureReasonTimeout. Would need to check if the ctx.Err() is context.DeadlineExceeded.

yeah agree - we could also get rid of p.timeoutChan and use the context to control the timeout, so instead of using p.timeoutChan = time.After(timeout), we inherit the parent context with a timeout context here

Thanks for pointing this out. I've removed the p.timeoutChan and replaced it with a deadline dependent context.

yyforyongyu · 2024-05-15T11:48:45Z

routing/payment_lifecycle.go

@@ -319,27 +322,42 @@ lifecycle:
 }

 // checkTimeout checks whether the payment has reached its timeout.


need to update the docs too

yyforyongyu · 2024-05-15T11:50:35Z

routing/payment_lifecycle.go

+	select {
+	case <-p.timeoutChan:
+		log.Warnf("payment attempt not completed before timeout")
+		err := failPayment(channeldb.FailureReasonTimeout)


can instead do return failPayment(...)

yyforyongyu · 2024-05-15T11:51:37Z

routing/payment_lifecycle.go

 	case <-p.router.quit:
 		return fmt.Errorf("check payment timeout got: %w",
 			ErrRouterShuttingDown)

-	// Fall through if we haven't hit our time limit.
+		// Fall through if we haven't hit our time limit.


think gofmt failed here

yyforyongyu · 2024-05-15T11:56:06Z

routing/payment_lifecycle.go

+			return err
+		}
+
+	case <-ctx.Done():


yeah agree - we could also get rid of p.timeoutChan and use the context to control the timeout, so instead of using p.timeoutChan = time.After(timeout), we inherit the parent context with a timeout context here

routing/payment_lifecycle.go

yyforyongyu · 2024-05-15T11:57:45Z

routing/payment_lifecycle.go

+		}
+
+	case <-ctx.Done():
+		log.Warnf("payment attempt context canceled")


Need cap the log and add a p.identifier.

yyforyongyu · 2024-05-15T11:58:32Z

routing/payment_lifecycle_test.go

@@ -179,6 +180,34 @@ func sendPaymentAndAssertFailed(t *testing.T,
 	}
 }

+// sendPaymentAndAssertFailed calls `resumePayment` and asserts that an error


Suggested change

// sendPaymentAndAssertFailed calls `resumePayment` and asserts that an error

// sendPaymentAndAssertContextCancelled calls `resumePayment` and asserts that an error

I've called it sendPaymentAndAssertError.

yyforyongyu · 2024-05-15T11:59:45Z

routing/payment_lifecycle_test.go

@@ -179,6 +180,34 @@ func sendPaymentAndAssertFailed(t *testing.T,
 	}
 }

+// sendPaymentAndAssertFailed calls `resumePayment` and asserts that an error
+// is returned.
+func sendPaymentAndAssertContextCancelled(t *testing.T,


looks like it's the same as sendPaymentAndAssertFailed?

I've consolidated the two.

yyforyongyu · 2024-05-15T12:30:50Z

routing/payment_lifecycle.go

+			return err
+		}
+
+		return ctx.Err()


Unfortunately we cannot return an error here, as it'd cause the payment loop to exit. For the timeout case, we never exit the loop. Consider this case, say a payment is made of two HTLC attempts,

HTLC1 ---> hop1 ---> remote HTLC2 ---> hop2 ---> remote

Then remote settles the invoice, but hop2 is offline, we'd end up in this case,

HTLC1 ---> hop1 ---> remote ---> settled HTLC2 ---> hop2 ---> remote ---> pending

After 60s, we'd time out this payment, but we won't quit the loop as we still think the payment is inflight,

lnd/channeldb/payment_status.go

Line 148 in 9d358bc

// | true | true | false | true | StatusInFlight |

Then hop2 comes online and settles it, we'd consider the payment successful.

lnd/channeldb/payment_status.go

Line 156 in 9d358bc

// | false | true | false | true | StatusSucceeded |

In other words, once the HTLC is sent, there's no easy way to cancel it unless you get a clear response from your peer saying it's failed/settled. If we exit here because the context is canceled, the payment would still have the status StatusInFlight.

Thank you for the detailed explanation. I see why we can't interrupt the entire payment loop but instead behave exactly how we do it the timeout case.

coderabbitai · 2024-05-16T17:07:59Z

Important

Auto Review Skipped

Auto reviews are disabled on this repository.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?

Share

Tips

Chat

There are 3 ways to chat with CodeRabbit:

Review comments: Directly reply to a review comment made by CodeRabbit. Example:
- I pushed a fix in commit <commit_id>.
- Generate unit testing code for this file.
- Open a follow-up GitHub issue for this discussion.
Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
- @coderabbitai generate unit testing code for this file.
- @coderabbitai modularize this function.
PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
- @coderabbitai generate interesting stats about this repository and render them as a table.
- @coderabbitai show all the console.log statements in this repository.
- @coderabbitai read src/utils.ts and generate unit testing code.
- @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (invoked as PR comments)

@coderabbitai pause to pause the reviews on a PR.
@coderabbitai resume to resume the paused reviews.
@coderabbitai review to trigger a review. This is useful when automatic reviews are disabled for the repository.
@coderabbitai resolve resolve all the CodeRabbit review comments.
@coderabbitai help to get help.

Additionally, you can add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.

CodeRabbit Configration File (`.coderabbit.yaml`)

You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
Please see the configuration documentation for more information.
If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

Visit our Documentation for detailed information on how to use CodeRabbit.
Join our Discord Community to get help, request features, and share feedback.
Follow us on X/Twitter for updates and announcements.

hieblmi · 2024-05-17T08:14:41Z

Thanks to the reviewers so far. I think I addressed all your concerns, namely:

Add a cacelable flag to the SendPaymentRequest that preserves the current behavior of a long running payment loop even if the stream context is canceled, but cancels the payment loop if set to true.
Wrap the send payment context into a deadline context in case the a timeout is set in the SendPaymentRequest.
Remove the timeout channel in payment_lifecycle.go in favor of the new timeout context.

I tested different scenarios locally and didn't encounter an error, additionally to the added test.

Looking forward to another round of feedback.

bitromortac · 2024-05-17T12:45:36Z

lnrpc/routerrpc/router.proto

+
+    /*
+    If set, the payment loop can be interrupted by manually canceling the
+    payment context, even before the payment timeout is reached.


Maybe add something like: Note that the payment may still succeed after cancellation, as in-flight attempts can still settle afterwards. Canceling will only prevent further attempts from being sent.. Similarly for the cli flag

bitromortac · 2024-05-17T13:15:25Z

lnrpc/routerrpc/router_server.go

+
+	// Cancel the timeout context. If the context already timed out or if
+	// there was no timeout provided, this will be a no-op.
+	cancel()


this doesn't cancel the payment directly because trackPayment is blocking?

This call is to avoid a linter issue that would arise if I wrote above ctx, _ = context.WithDeadline(ctx, timeout). And if I assign the cancel I have to call it. The right place seems to be after trackPayment because then the context can be canceled.

an alternative could be to place it after sendPayment in SendPaymentAsync, but not sure that's a good pattern

could also silence the linter here with a comment as to why we don't need the context in this case.

bitromortac · 2024-05-17T13:27:14Z

routing/payment_lifecycle.go

+// checkContext checks whether the payment context has been canceled.
+// Cancellation occurs manually or if the context times out.
+func (p *paymentLifecycle) checkContext(ctx context.Context) error {
+	failPayment := func(reason channeldb.FailureReason) error {


I think this can be inlined again

bitromortac · 2024-05-17T13:28:34Z

routing/payment_lifecycle.go

+	case <-ctx.Done():
+		failureReason := channeldb.FailureReasonError
+
+		switch {


This could be an if/else instead

In this commit we set up the payment loop context according to user-provided parameters. The `cancelable` parameter indicates whether the user is able to interrupt the payment loop by cancelling the server stream context. We'll additionally wrap the context in a deadline if the user provided a payment timeout. We remove the timeout channel of the payment_lifecycle.go and in favor of the deadline context.

hieblmi self-assigned this May 7, 2024

hieblmi added this to the v0.18.1 milestone May 7, 2024

hieblmi added the routing label May 7, 2024

hieblmi force-pushed the cancel-estimateroutefee branch 2 times, most recently from 48cfd5b to 1d6e008 Compare May 7, 2024 12:15

hieblmi commented May 7, 2024

View reviewed changes

routing/payment_lifecycle.go Outdated Show resolved Hide resolved

hieblmi force-pushed the cancel-estimateroutefee branch 2 times, most recently from dcdd913 to 02072fa Compare May 7, 2024 12:50

hieblmi changed the title ~~routing: Cancelable payment loop~~ routing: cancelable payment loop May 7, 2024

hieblmi added the payments Related to invoices/payments label May 7, 2024

hieblmi force-pushed the cancel-estimateroutefee branch from 02072fa to e2cfd30 Compare May 8, 2024 09:53

hieblmi commented May 8, 2024

View reviewed changes

lnrpc/routerrpc/router_server.go Show resolved Hide resolved

hieblmi force-pushed the cancel-estimateroutefee branch from e2cfd30 to 4fa701d Compare May 8, 2024 12:28

hieblmi requested a review from bitromortac May 8, 2024 15:02

ellemouton reviewed May 9, 2024

View reviewed changes

routing/payment_lifecycle.go Outdated Show resolved Hide resolved

routing/payment_lifecycle.go Outdated Show resolved Hide resolved

hieblmi force-pushed the cancel-estimateroutefee branch 2 times, most recently from 0bfb3db to 2ab2bd9 Compare May 10, 2024 09:15

ellemouton approved these changes May 10, 2024

View reviewed changes

hieblmi force-pushed the cancel-estimateroutefee branch from 2ab2bd9 to 4f18995 Compare May 10, 2024 09:51

lightningnetwork deleted a comment from coderabbitai bot May 10, 2024

yyforyongyu reviewed May 11, 2024

View reviewed changes

hieblmi mentioned this pull request May 13, 2024

loopout: configurable payment timeout for off-chain payments lightninglabs/loop#743

Open

1 task

bitromortac reviewed May 13, 2024

View reviewed changes

bhandras reviewed May 14, 2024

View reviewed changes

yyforyongyu reviewed May 15, 2024

View reviewed changes

hieblmi mentioned this pull request May 16, 2024

routing: log edge when skipping it #8765

Merged

hieblmi force-pushed the cancel-estimateroutefee branch from 4f18995 to 215b4c9 Compare May 16, 2024 14:44

lightningnetwork deleted a comment from coderabbitai bot May 16, 2024

hieblmi force-pushed the cancel-estimateroutefee branch from 215b4c9 to 9a3fa63 Compare May 16, 2024 15:25

hieblmi force-pushed the cancel-estimateroutefee branch from 9a3fa63 to 99567f9 Compare May 16, 2024 16:55

lightningnetwork deleted a comment from coderabbitai bot May 16, 2024

hieblmi force-pushed the cancel-estimateroutefee branch from 99567f9 to d15764d Compare May 16, 2024 17:07

routing: fix typos and wrap errors

21bf0f4

hieblmi force-pushed the cancel-estimateroutefee branch 3 times, most recently from 614296b to 83eb67a Compare May 17, 2024 08:08

hieblmi requested review from bhandras, bitromortac and yyforyongyu May 17, 2024 08:09

bitromortac reviewed May 17, 2024

View reviewed changes

hieblmi added 2 commits May 17, 2024 17:30

lnrpc: cancelable sendpayment request

5fb505d

hieblmi force-pushed the cancel-estimateroutefee branch from 83eb67a to 09592bf Compare May 17, 2024 15:38

lncli: cancelable flag for SendPaymentRequest

09592bf

hieblmi requested a review from bitromortac May 17, 2024 15:39

		@@ -319,27 +322,42 @@ lifecycle:
		}

		// checkTimeout checks whether the payment has reached its timeout.

	// sendPaymentAndAssertFailed calls `resumePayment` and asserts that an error
	// sendPaymentAndAssertContextCancelled calls `resumePayment` and asserts that an error

routing: cancelable payment loop #8734

Are you sure you want to change the base?

routing: cancelable payment loop #8734

Conversation

hieblmi commented May 7, 2024 • edited

ellemouton left a comment

Choose a reason for hiding this comment

ellemouton left a comment

Choose a reason for hiding this comment

yyforyongyu left a comment

Choose a reason for hiding this comment

hieblmi commented May 11, 2024

bitromortac left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

coderabbitai bot commented May 16, 2024

Auto Review Skipped

Chat

CodeRabbit Commands (invoked as PR comments)

CodeRabbit Configration File (.coderabbit.yaml)

Documentation and Community

hieblmi commented May 17, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hieblmi commented May 7, 2024 •

edited

CodeRabbit Configration File (`.coderabbit.yaml`)

hieblmi commented May 17, 2024 •

edited