Intercept forward htlc #4018

roeierez · 2020-02-19T17:30:46Z

This PR introduces the ability to intercept htlc forward events that are handled at the htlcswitch package and to override the switch default behavior. Basically, similar to HODL invoices, this PR enables routing nodes to HODL HTLCs and to run custom validations or processes before forwarding (or cancelling) the HTLC.
More specifically for every HTLC that is intended to be forwarded, the implementor can choose to hold the forward and signal the switch to wait for a later resolution. Such resolution can be one of the following:

Resume - Resume execution to the default behavior (probably just forward).
Fail - Fail the htlc backward, the implementor is not interested in forwarding it.
Settle - For a given preimage that was revealed in a way not known to the switch the implementor can settle the htlc.
Having more control over the forward behavior enables new important use cases to be implemented.
We are specifically looking to support the following use cases:

Pay to Open
This is about being able to pay someone with no channel or with a low capacity channel by opening a channel during the payment.
Let's say Alice (that has a channel) wants to pay Bob (that doesn't have a channel).

Bob is connected to a node and issues an invoice with a routing hint that points to a fake channel between Bob and the node.
When Alice pays Bob's invoice, the node intercepts the HTLC and holds it.
At this stage one of the two options can follow:
- The node opens a channel to Bob and upon confirmation (or not) resumes the forward.
- The node opens a channel to Bob, pays Bob the original Alices amount minus service fee (in a different invoice), gets the preimage from Bob (in exchange to the payment) and settles the hold forward.

Notify to Pay
Let's say Alice wants to pay Bob, a mobile user, which is connected to a node via private channel.

Alice pays, the node intercept the forward, checks Bob's availability and notices Bob is offline (inactive).
The node notifies Bob using an agreed-upon notification mechanism.
The node waits for Bob to be online and once the channel is active, it resumes the forward.

Implementation flow & Usage

Link forward a packet, packet intercepted and sent to the RPC layer.
For each packet the RPC layer do the following:

Hold the packet immediately and signal hold backwards to the switch.
Send the packet for the client

In parallel RPC listens to client resolutions responses on the same stream and resolve packets accordingly.
On disconnection all remaining held packets are resolved using the default behavior (currently Resume)

I am attaching a sample file to show a sample usage that filter packets according to amount
cmd_intercept_forward.txt

halseth · 2020-02-20T09:08:56Z

Can be integrated with the HTLC notifier? #3781
cc @carlaKC

carlaKC · 2020-02-20T11:22:18Z

Can be integrated with the HTLC notifier? #3781

I think they're different sides of the same coin? HTLCNotifier notifies forwarding decisions post-fact, once we have made the forwarding/failing decision.

My understanding of this is that it's preemptive: receive forward, callout to external forwarding predicate and then return, as specified in #3953.

roeierez · 2020-02-20T11:37:45Z

Can be integrated with the HTLC notifier? #3781

@halseth thanks for bringing this to the discussion.
Some things to note when thinking about integration:

The notifications are dispatched after the forward has been processed while the interception should be hooked before that, this means the current htlcNotifier interface needs to be extended.
Interception is bidirectional and need to handle responses as well, this affects both the middleware and the RPC layer.
Controlling the forwards will probably need different RPC permissions than getting htlc notifications.
One example I see in the code that seems similar to this case is the channel events vs the channel acceptor. Two different mechanisms that have similar path in the code.

I think it will be an attempt to mix different things together.
Hope that helps.

Roasbeef · 2020-02-21T02:20:27Z

Interception should be done at the switch, not the link. Also extreme care must be taken to not block the entire forwarding fabric, as the switch manages its state synchronously atm.

champo · 2020-02-21T19:49:38Z

@roeierez I really like the look of the API here. I've also been working on something similar. It allows you to register virtual channels and control HTLCs there through gRPC. It adds a new Link implementation and the intercept happens there. As far as the switch is concerned, nothing changes from a regular channel, which I think goes along the lines of what @Roasbeef suggests.

The highly-WIP code is at b3b2f7f and an usage example at https://gist.github.com/champo/8e90ce1ef38f2dd546814ff7898988c3 A knonw issue is that it doesn't close circuits so it kinda stops working after a success settle. It's easy to fix but I haven't gotten around to it yet.

roeierez · 2020-02-25T09:20:38Z

Interception should be done at the switch, not the link. Also extreme care must be taken to not block the entire forwarding fabric, as the switch manages its state synchronously atm.

I have moved the interception to the switch, added unit tests and the Router RPC implementation.

champo · 2020-03-01T23:51:18Z

lnrpc/routerrpc/router.proto

+    uint64 htlc_id = 2;
+}
+
+message ForwardHtlcInterceptRequest {


It'd be great to include the sphinx packet in this message. It would allow a receiver to check amount and expiry as it would if the HTLC had been forwarded through a link. This is useful for reverse swaps.

@champo thanks for the feedback.
Currently the fields amount_sat expiry are what you are looking for as they are the actual values that would have been used in the forwarded packet in the default switch behavior.

A third-party receiver (say a mobile app) shouldn't trust the fields. They would want to check the information against the sphinx. It would also enable usage of new fields like payment secret which is only known to the receiver.

@champo I agree. At first I exposed only the fields that are required for the use cases I mentioned in the PR description. I do think it makes sense to expose all the forwarding info (such as outgoing channel, tlv fields and raw sphinx packet) as other use cases may need this info in order to take a decision at interception time.

It'd be great to include the sphinx packet in this message.

The raw packet itself can't be verified unless the receiver has the private key for the end node.

@Roasbeef in our case, the end node and the interceptor communicate over HTTP to negotiate the settling of the HTLC. The end node can then decode the sphinx packet to reduce trust.

htlcinterceptor/middleware.go

htlcswitch/switch.go

Roasbeef

LGTM 🔱

Don't think we need the REST annotation since this type of RPC isn't supported atm. Should be ready to land after a rebase!

Roasbeef · 2020-06-18T02:57:15Z

htlcswitch/interceptable_switch.go

+
+	switch htlc := packet.htlc.(type) {
+	case *lnwire.UpdateAddHTLC:
+		// We are not interested in intercepting initated payments.


We might want to reverse this in the future, as it would allow for the interceptor to be used as a firewall for all outgoing send attempts.

Roasbeef · 2020-06-18T02:59:37Z

lnrpc/rest-annotations.yaml

@@ -211,6 +211,8 @@ http:
      # deprecated, no REST endpoint
    - selector: routerrpc.Router.TrackPayment
      # deprecated, no REST endpoint
+    - selector: routerrpc.HtlcInterceptor


I don't think we need anything here since this is a bi directional streaming RPC, which we don't yet support. The current web sockets implementation only supports server-side streaming requests.

I added it because it is required by the make rpc-check

As part of the preparation to the switch interceptor feature, this function is changed to return error instead of error channel that is closed automatically. Returning an error channel has become complex to maintain and implement when adding more asynchronous flows to the switch. The change doesn't affect the current behavior which logs the errors as before.

roeierez · 2020-06-18T10:00:05Z

Don't think we need the REST annotation since this type of RPC isn't supported atm

The build requires it atm: make rpc-check

ready to land after a rebase!

Rebased.

joostjager

reviewed 2b0bdab

joostjager · 2020-06-18T11:20:04Z

lnrpc/routerrpc/forward_interceptor.go

+		if err != nil {
+			return err
+		}
+		err = interceptedForward.Settle(preimage)


return interceptedForward.Settle(preimage)

joostjager · 2020-06-18T11:21:28Z

lnrpc/routerrpc/router_server.go

+// forwards and find them when manual resolution is later needed.
+func (s *Server) HtlcInterceptor(stream Router_HtlcInterceptorServer) error {
+	// We ensure there is only one interceptor at a time.
+	if !atomic.CompareAndSwapInt32(&s.forwardInterceptorActive, 0, 1) {


I think it is still useful to make sure that any user of interceptable switch uses it correctly. But non-blocking

In this commit we implement a wrapper arround the switch, called InterceptableSwitch. This kind of wrapper behaves like a proxy which intercepts forwarded packets and allows an external interceptor to signal if it is interested to hold this forward and resolve it manually later or let the switch execute its default behavior. This infrastructure allows the RPC layer to expose interceptor registration API to the user and by that enable the implementation of custom routing behavior.

In this commit we add the ability to intercept forwarded htlc packets straight from the RPC layer. The RPC layer handles a bidrectional stream that comminucates to the client the intercepted packets and handles its response by coordinating with the interceptable switch.

hsjoberg · 2020-08-26T08:15:00Z

lnrpc/routerrpc/router.proto

+    Forwarded HTLC requests are sent to the client and the client responds with
+    a boolean that tells LND if this htlc should be intercepted.
+    In case of interception, the htlc can be either settled, cancelled or
+    resumed later by using the ResolveHoldForward endpoint.


Hi @roeierez. This description text looks wrong. I don't find any ResolveHoldForward endpoint.
AFAICT how to resume the incoming payment is by sending with action set to RESUME in this gRPC stream.
Perhaps I'm missing something?

Also by reading the protobuf, it doesn't look lke the client is supposed to respond with a boolean if the HTLC should be intercepted or not (rather it is decided by the incoming_circuit_key?)

@hsjoberg You are right. Unfortunately this description was left from a previous iteration and is indeed wrong.
The interface was changes and the client should return a ForwardHtlcInterceptResponse with action that maps to one of the follows:

SETTLE: to settle the htlc (a preimage is expected as well).
FAIL: to fail the htlc
RESUME: to resume with default behavior.

@roeierez Thanks, that makes sense.

Cheers!

hsjoberg · 2020-09-14T15:01:40Z

@roeierez Correct me if I'm wrong about this, but AFAICT as it stands right now, this API is dangerous if an amountless invoice is used (and action is SETTLE).
Only the receiving node knows about amt_to_forward and payment_secret in their TLV payload, so isn't there a vulnerability here where the LSP node could give up the preimage for too little satoshi? (Basically the amountless invoice problem again.)

EDIT: Below is based on misunderstandings from my side.

roeierez · 2020-09-14T15:22:19Z

@roeierez Correct me if I'm wrong about this, but AFAICT as it stands right now, this API is dangerous if an amountless invoice is used (and action is SETTLE).
Only the receiving node knows about amt_to_forward and payment_secret in their TLV payload, so isn't there a vulnerability here where the LSP node could give up the preimage for too little satoshi? (Basically the amountless invoice problem again.)

@hsjoberg Not following. Settling an htlc backwards requires a preimage which is passed to the interceptor by the next node in the path either using regular lightning mechanism or some custom swap logic. In any case interceptor can make sure it has the required funds before settling. Don't see any risk.

hsjoberg · 2020-09-14T16:21:44Z

@roeierez In the case of an undetermined amount ("amountless invoice"), the interceptor node must somehow know how much it should reveal the preimage for to settle the payment. AFAICT basically what feature bits 14/15 is for.

roeierez · 2020-09-14T17:46:07Z

@roeierez In the case of an undetermined amount ("amountless invoice"), the interceptor node must somehow know how much it should reveal the preimage for to settle the payment. AFAICT basically what feature bits 14/15 is for.

@hsjoberg The interceptor doesn't know the total amount of a payment regardless of amount-less invoice or not. Invoice is between the payer and the payee. Also the way the interceptor gets the pre-image to settle backwards is totally implementation dependent so in that sense it can be risky or not.
The implementation in Breez won't change the way lightning works, means the interceptor will get the pre-image from the payee (once his invoice is settled) so I don't see how this adds risk to the interceptor node.

hsjoberg · 2020-09-14T20:22:39Z

The interceptor doesn't know the total amount of a payment regardless of amount-less invoice or not. Invoice is between the payer and the payee

But that could just be trivially communicated with the interceptor somehow in the case of a normal invoice with amount.

@roeierez Sorry, I will try to explain my concern better. I'll take it from the beginning:
There exists a vulnerability with amountless invoices where a malicious party Mallory could attempt to make the receiver reveal the preimage by forwarding another payment with same hash but with less amount.

Let's say A wants to pay C 1 BTC through M:
A -> M -> C

Instead of forwarding the real payment, M could make a new payment with 1 sat using the same hash in an attempt lure C to reveal the preimage.

This issue has been demonstrated in the past and fixed with the 14/15 payment secret and/or enforcing TLV/MPP/keysend. My high-level understanding is that a nonce is introduced that the payer is sending to the payee, which can be used to find out tampering.
The payment secret is encrypted in a TLV for the receiving node.

Now let's say we're using Pay to Open, and interceptor settling the payment and opening channel to C (either paying through push amount or another invoice, shouldn't matter AFAICT).
A wants to pay C 1 BTC through M and interceptor node I. C has no inbound channel:
A->M->I->C

The malicious party M could attempt to lure I and thus also C by forwarding 1 sat instead of the real payment.
Now, this is the gut of the whole thing, and this is more of a question rather than a conclusion, but even if the new payment scheme using TLV is enforced (or 14 payment secret), lnd and the code as it stands right now wouldn't check for tampering because it won't decrypt and check the actual receiving nodes (C) packet (as I is just settling it and not letting C see his packet), thus it looks vulnerable to the old amountless invoice problem.

roeierez · 2020-09-14T21:01:04Z

But that could just be trivially communicated with the interceptor somehow in the case of a normal invoice with amount.

Then what prevents the payee to communicate the amount he expects in the zero invoice case as well?

as I is just settling it and not letting C see his packet

If this is the case how the interceptor has the pre-image? It seems that you described a flow where the interceptor has the pre-image beforehand and this is not how we (at Breez) implementing Pay to Open. We do forward the packet to the destination and only once the destination reveals the pre-image the interceptor settles backwards using regular lightning mechanism.
The interceptor, in our implementation, also agrees with the end node on the amount to forward and the LSP fee (for opening a.channel) beforehand.
So in fact the interceptor has all the means to hold the forward until he receives enough htlcs amount to take his fees, before resuming the forwards.

hsjoberg · 2020-09-14T21:57:39Z

Then what prevents the payee to communicate the amount he expects in the zero invoice case as well?

The payee doesn't know what amount to expect, hence amountless invoice.
The final TLV payload has to be read by either the interceptor or payee somehow, as there lies the payment_secret and amt_to_forward that the payer sent to the payee.

If this is the case how the interceptor has the pre-image? It seems that you described a flow where the interceptor has the pre-image beforehand

The wallet simply hands the preimage to the LSP and the LSP uses the SETTLE action via the HtlcIntercept API.
The flow I am thinking about is not where the interceptor continues the forwarding (RESUME), rather it settles it themselves (SETTLE), and then afterwards opens a channel with the wallet.
In short:

Wallet creates an invoice with route hint which goes to the LSP node with fake short chan id.
Wallet hands preimage to LSP
Payer pays invoice and uses the fake short chan id provided in route hint
LSP intercepts the incoming HTLC and SETTLE's it
LSP opens a new channel to Wallet (and pays using push amount, another invoice or something else)

I explained briefly in the OP that the concern is about the SETTLE action.

this is not how we (at Breez) implementing Pay to Open

I'm not talking about Breez's implementation per se, just resuming the forward should be completely fine.
But I think it should apply to the 3.2 scenario in Pay to Open described in your OP:
The node opens a channel to Bob, pays Bob the original Alices amount minus service fee (in a different invoice), gets the preimage from Bob (in exchange to the payment) and settles the hold forward.

The interceptor requests a new invoice from the wallet to pay through and then settles the hold forward in this scenario(?). Thus the original sender's TLV payload would not reach the receiver.

So in fact the interceptor has all the means to hold the forward until he receives enough htlcs amount to take his fees, before resuming the forwards.

Yes, if the interceptor just resumed the forwarding and uses a newly created channel to the payee, the problem I've described does not apply as 14 payment secret (or any other method) can be enforced.

roeierez requested review from cfromknecht and Roasbeef as code owners February 19, 2020 17:30

carlaKC added htlcswitch P3 might get fixed, nice to have routing nodes labels Feb 20, 2020

Roasbeef added this to the 0.11.0 milestone Feb 21, 2020

roeierez force-pushed the intercept-forward-htlc branch 8 times, most recently from df7f3de to a199479 Compare February 24, 2020 19:49

champo reviewed Mar 1, 2020

View reviewed changes

andrerfneves mentioned this pull request Mar 3, 2020

[WIP] Add post for socratic 102 BitDevsNYC/BitDevsNYC.github.io#50

Merged

roeierez force-pushed the intercept-forward-htlc branch from a199479 to b1cdbc7 Compare March 4, 2020 11:24

roeierez changed the title ~~Intercept forward htlc [WIP]~~ Intercept forward htlc Mar 4, 2020

champo reviewed Mar 11, 2020

View reviewed changes

roeierez force-pushed the intercept-forward-htlc branch 2 times, most recently from 5d72a87 to c416106 Compare March 29, 2020 15:35

roeierez requested a review from joostjager as a code owner March 29, 2020 15:35

roeierez force-pushed the intercept-forward-htlc branch from c416106 to ef5113a Compare March 29, 2020 15:50

cfromknecht added this to In progress in v0.11.0-beta via automation Apr 21, 2020

roeierez force-pushed the intercept-forward-htlc branch from b95122a to f2faff9 Compare June 16, 2020 15:54

Roasbeef approved these changes Jun 18, 2020

View reviewed changes

roeierez force-pushed the intercept-forward-htlc branch from f2faff9 to cfa2ee8 Compare June 18, 2020 09:52

roeierez force-pushed the intercept-forward-htlc branch from cfa2ee8 to 2b0bdab Compare June 18, 2020 10:04

v0.11.0-beta automation moved this from Review in progress to Reviewer approved Jun 18, 2020

joostjager approved these changes Jun 18, 2020

View reviewed changes

roeierez force-pushed the intercept-forward-htlc branch from 2b0bdab to 5501da3 Compare June 18, 2020 12:13

roeierez force-pushed the intercept-forward-htlc branch from 5501da3 to 7b56268 Compare June 18, 2020 12:21

joostjager merged commit 8f2a2fc into lightningnetwork:master Jun 22, 2020

v0.11.0-beta automation moved this from Reviewer approved to Done Jun 22, 2020

joostjager mentioned this pull request Jun 23, 2020

htlcswitch: expose additional data for intercepted packets #4405

Merged

roeierez mentioned this pull request Jun 28, 2020

Skip funding confirmation #4424

Closed

Roasbeef mentioned this pull request Jul 16, 2020

htlc callback acceptance #3953

Closed

hsjoberg reviewed Aug 26, 2020

View reviewed changes

hsjoberg mentioned this pull request Sep 10, 2020

Allow user specify routing hints in private invoice. #3672

Merged

guggero mentioned this pull request Nov 30, 2020

Expose payment hash to the API for forwarded payments #2731

Closed

Roasbeef mentioned this pull request Aug 31, 2021

Implement streaming RPC functionality for remote and local balance updates in a node's channels #2734

Closed

yyforyongyu mentioned this pull request Aug 31, 2021

lnrpc: add payment hash to forwarding event #4759

Closed

joostjager mentioned this pull request Nov 30, 2021

On-chain resolution flow ignores htlc interceptor settles #6040

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Intercept forward htlc #4018

Intercept forward htlc #4018

roeierez commented Feb 19, 2020 •

edited

halseth commented Feb 20, 2020

carlaKC commented Feb 20, 2020

roeierez commented Feb 20, 2020 •

edited

Roasbeef commented Feb 21, 2020

champo commented Feb 21, 2020

roeierez commented Feb 25, 2020

champo Mar 1, 2020

roeierez Mar 2, 2020

champo Mar 2, 2020

roeierez Mar 5, 2020

Roasbeef May 13, 2020

champo Jun 9, 2020

Roasbeef left a comment

Roasbeef Jun 18, 2020

Roasbeef Jun 18, 2020

roeierez Jun 18, 2020

roeierez commented Jun 18, 2020

joostjager left a comment •

edited

joostjager Jun 18, 2020

roeierez Jun 18, 2020

joostjager Jun 18, 2020

hsjoberg Aug 26, 2020

roeierez Aug 26, 2020

hsjoberg Aug 26, 2020

hsjoberg commented Sep 14, 2020 •

edited

roeierez commented Sep 14, 2020

hsjoberg commented Sep 14, 2020 •

edited

roeierez commented Sep 14, 2020

hsjoberg commented Sep 14, 2020 •

edited

roeierez commented Sep 14, 2020

hsjoberg commented Sep 14, 2020 •

edited

Intercept forward htlc #4018

Intercept forward htlc #4018

Conversation

roeierez commented Feb 19, 2020 • edited

halseth commented Feb 20, 2020

carlaKC commented Feb 20, 2020

roeierez commented Feb 20, 2020 • edited

Roasbeef commented Feb 21, 2020

champo commented Feb 21, 2020

roeierez commented Feb 25, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Roasbeef left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

roeierez commented Jun 18, 2020

joostjager left a comment • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hsjoberg commented Sep 14, 2020 • edited

roeierez commented Sep 14, 2020

hsjoberg commented Sep 14, 2020 • edited

roeierez commented Sep 14, 2020

hsjoberg commented Sep 14, 2020 • edited

roeierez commented Sep 14, 2020

hsjoberg commented Sep 14, 2020 • edited

roeierez commented Feb 19, 2020 •

edited

roeierez commented Feb 20, 2020 •

edited

joostjager left a comment •

edited

hsjoberg commented Sep 14, 2020 •

edited

hsjoberg commented Sep 14, 2020 •

edited

hsjoberg commented Sep 14, 2020 •

edited

hsjoberg commented Sep 14, 2020 •

edited