Proposal: Connectors should forward (not deliver) ILP packets with amounts equal to the max uint64

[emschwartz]

This proposes that connectors SHOULD (or MUST?) skip the logic for delivering the exact destination amount when the amount in the ILP packet is the maximum value for unsigned 64-bit integers (18446744073709551615).

Background on Forwarding vs Delivery

Forwarding: Normally, connectors forward payments by applying their exchange rate to the incoming transfer's amount to get the outgoing transfer's amount.

Delivery: When the ilp-connector thinks it is the last connector in a payment path, it will attempt to deliver the exact amount specified in the ILP packet to the receiver (code). If the incoming transfer amount is higher than necessary to deliver the destination amount, the connector will pocket the difference and deliver the amount from the ILP packet. If the incoming transfer amount is too low, the connector will reject the payment with an R01: Insufficient Source Amount error.

Assumptions Underpinning the Delivery Behavior

(These are all debatable)

• Overpayment is considered a bug, rather than a feature. If a store requests $10 it is problematic for $10.05 to arrive. Note that the ilp client is currently set to disallow overpayment by default (code) and will reject incoming payments if the transfer amount is even slightly over the amount in the ILP packet.
• (Underpayment:) Payments where the amount delivered to the receiver is less than the amount in the ILP packet are likely to be rejected anyway, so the connector might as well reject them early and avoid putting funds on hold for nothing.
• The connector assumes that it is the final hop if its account shares the ledger prefix with the destination address in the ILP packet. (For example, if the connector is example.ledger.connector and the packet is addressed to example.ledger.bob or example.ledger.bob.something.else, the connector will send the exact amount in the ILP packet.)
• If there are additional segments in the ILP address after the account (the .something.else in the example above), there cannot be connectors with exchange rates other than 1:1 between the account on the connector's ledger and the receiver. (The idea is that the segments after the address could represent subaccounts or "subledgers", but they must be denominated in the same units as the bigger ledger.)

Problems with Exact Delivery

The sender may not want connectors to try to deliver the exact amount in the ILP packet if:

1. The sender didn't know exactly how much would arrive at the destination but wanted to send a fixed source amount anyway
2. The sender has an out-of-band communication channel with the receiver through which they can discuss how much was received or how much should have been received
3. The sender is streaming payments and wants as much to arrive as possible for each one
4. There is another connector that applies a non-1:1 rate after the connector that assumes it is the last (for example if the subledger uses a smaller scale than the ledger, so the amount in the ILP packet would seem too large)

Proposal: Special-Case ILP Packet Amounts of MAX_UINT64

Problems 1-3 would be solved if connectors would simply skip the usual logic for trying to deliver the exact amount when the ILP packet amount is 18446744073709551615. In most cases, the last connector would continue to deliver the exact destination amount. However, when the sender specifically does not want the connector to try to do this, they would have a way to indicate that. Connectors would forward the payment using their normal rate, rather than delivering the amount from the packet or rejecting the incoming transfer with an error.

This proposal would not address problem 4, which would only be solved by entirely removing the delivery behavior. We previously decided that it was more important to include the exact delivery to avoid overpayment than to avoid this type of subledgering.

Alternative Solutions

Connectors Shouldn't Reject for Underpayment

Problems 1-3 mainly stem from the connector rejecting payments when the transfer amount is lower than it would need to be to deliver the exact destination amount in the ILP packet. Instead of special-casing the value 18446744073709551615, connectors could simply forward all payments where the source amount appears to be too small. It would then be up to the receiver to reject the payment if it does not meet their criteria.

The main downside of this approach is that there may be a good chance that the receiver will reject the payment anyway, so the connector(s) will have put their funds on hold for nothing. If the sender more intentionally communicates that the connector should skip the delivery logic, the assumption would be that the receiver would be prepared to handle the payment.

Flag in the ILP Packet

We could use the ASN.1 extensions feature in the ILP packet to add a flag that the sender can use to explicitly communicate "don't try to deliver the exact amount", rather than effectively using the max amount value as this flag. The downside of doing this is a) that we have to use ASN.1 extensions (which are a bit complicated/annoying to implement) and b) it's a change to the ILP packet (which we'd like to avoid as much as possible).

[michielbdejong]

Normally, connectors forward payments by applying their exchange rate to the incoming transfer's amount to get the outgoing transfer's amount.

That may be true of ilp-connector, but in general, it's not a rational gain-maximizing strategy, right? Wouldn't it be smarter for a forwarding connector to always try to minimize the amount in the outgoing transfer?

[michielbdejong]

Proposal: Special-Case ILP Packet Amounts of MAX_UINT64

I see the reason for this special case (as a sender you don't want to specify a specific destination amount), but I would formulate it differently.

On the Interledger level, we can just leave out the amount from the ILP packet. And I know what you're going to say: that makes it impossible to construct the packets OER encoding. True, and that poses two problems: you need a way to deal with that when determining the PSK input, and you need a way to deal with it when determining the value of the transfer.ilp field in the LPI.

The way we deal with amount-less ILP payments would be to generate the PSK input, as well as the transfer.ilp value, as if it was an ILP payment with a MAX_INT amount. But on the Interledger level, I would say the destination amount is undefined, not that the destination amount has a value of 2^64-1.

[sappenin]

Flag in the ILP Packet

We could use the ASN.1 extensions feature in the ILP packet to add a flag that the sender can use to explicitly communicate "don't try to deliver the exact amount", rather than effectively using the max amount value as this flag. The downside of doing this is a) that we have to use ASN.1 extensions (which are a bit complicated/annoying to implement) and b) it's a change to the ILP packet (which we'd like to avoid as much as possible).

Despite the difficulty in adjusting the ILP packet, I would prefer being overt about the "do not deliver" functionality proposed in this issue. Using an indirect-message (i.e., the magic number 18446744073709551615) just feels icky to me (I wish I could be more technical in that statement, but the best concrete argument I can make is that an implementation might overlook the magic treatment, or what if somebody actually wanted to sent 18446744073709551615 units of something...but these are all pretty well handled in your #311 proposal).

What if we said said we would add a special flag to the ILP packet in ILP v2.0, or v1.1? To that end, would it make sense to extend the ILP packet with some sort of version flag so that implementations could easily determine which version of an ILP packet they're dealing with -- and if there's no "version", assume v1?

An while we're at, shouldn't there also be some sort of "feature-flag" capability in the ILP packet, for this very type of functionality change/enhancement/etc?

[emschwartz]

That may be true of ilp-connector, but in general, it's not a rational gain-maximizing strategy, right? Wouldn't it be smarter for a forwarding connector to always try to minimize the amount in the outgoing transfer?

Maybe. If the sender got a quote before (either using ILQPv1 or v2) you'd expect the source amount they're sending to be pretty close to the minimum. Also, if you try to take slightly more than your normal rate in the hopes that enough will still get to the receiver for it to be accepted, you may also run the risk of bringing down your success rating. In the long-run it may be better and simpler to just set an exchange rate that works for you and apply it every time.

On the Interledger level, we can just leave out the amount from the ILP packet. And I know what you're going to say: that makes it impossible to construct the packets OER encoding.

I'm not sure I understand this suggestion. ILP packets are currently defined as being the OER-encoding of the specified ASN.1 format. When you say "leave out" the amount, do you mean a) simply start encoding ILP packets as just the account and data b) redefine the ILP packet format to not include the amount or c) redefine the ILP packet format to make the amount optional (using an existence bitmap)?

version flag so that implementations could easily determine which version of an ILP packet they're dealing with

The ILP packets have a wrapper that includes a type byte. That would be used to add an Interledger Protocol Payment v2 packet.

Since we do have the ASN.1 extensions thing in the Interledger Protocol Payment packet, we might be able to add a version 1.1 without breaking current implementations.

just feels icky to me...What if we said said we would add a special flag to the ILP packet in ILP v2.0, or v1.1?

Totally reasonable suggestion. My main not-at-all-scientific objection to making a new version of the ILP packet format is that I really don't want us to make any more breaking changes at that layer now and I'm not sure we could get away with a version 1.1 without breaking stuff in practice. But it's fair to say that redefining the max amount is just a hacky way of making such a change that connectors need to support, which makes it effectively a v1.1 of the semantics, if not of the encoding format.

[michielbdejong]

redefining the max amount is just a hacky way of making such a change that connectors need to support

Yes, to introduce this behavior you should define a new ILP packet type. Just using the extensions would not be correct, since there may be connectors along the path that ignore your extension.

[justmoon]

After reading this proposal, I think it's worth thinking about deprecating the amount field entirely. We may not want to change the ILP packet format, but if we are introducing new semantics, I'd prefer if they were simpler than the old semantics. Getting rid of the forward vs delivery semantics in particular I think would be a worthy goal. I've tried to explain this concept to a bunch of people and it's always one of the hardest parts of ILP for people to wrap their heads around.

[adrianhopebailie]

I think this proposal has a lot of merit but would be improved by using an amount of zero (not MAX_UINT64). This is also how I would solve for @justmoon 's proposal in #312.

In #312 (comment) @emschwartz said:

I picked the max value for #311 because you might actually have a reason to deliver 0, whereas it seems much less likely that you would try to deliver 18446744073709551615 units. If connectors do any type of delivery it makes more sense to me that they would round the number down but that a max value would mean something like "as much as you can get there"

I think this is a red-herring. What reason would anyone ever have to deliver an amount of zero? In fact, what does delivery of an amount of zero even mean (if different from one of the behaviors below)?

If an ILP packet has an amount of zero it will be forwarded until one of the following happens (assuming existing implementations don't simply reject an amount of zero outright):

1. The connector forwarding the packet had too little in the incoming transfer to make an outgoing transfer of ANY size (the connector may some fixed costs). This connector will then reject the payment with an Insufficient Source Amount error (R01).
2. The receiver gets the packet and rejects the payment with an Insufficient Destination Amount error (F04)
3. The receiver gets the packet and rejects the payment because it's unexpected or the amount it received in the incoming transfer is too high.
4. The receiver gets the packet and produces the fulfillment

Getting rid of the forward vs delivery semantics in particular I think would be a worthy goal. I've tried to explain this concept to a bunch of people and it's always one of the hardest parts of ILP for people to wrap their heads around.

I don't agree that we can (or should) get rid of the forward vs delivery semantics. They are essential if you need to deliver an exact amount to a receiver (which I'm sure constitutes the majority of cases). The fact that it is hard to explain doesn't seem like a good reason to make a change that prevents us executing the most common use case.

[emschwartz]

I was convinced that 0 would be better to use than the max uint64 value (and have updated #309 to use 0). However, I agree with @justmoon that we should get rid of the delivery idea in general. Let's continue that discussion in #312