RFC-3: Remove amount field

[justmoon]

Through @emschwartz's work on the ILQPv2 protocol, it became apparent that there are use cases where we would like the connector to forward payments according to their offered rate. That is, we want connectors to ignore the amount field in the ILP packet.

The main reason there is an amount field in the ILP packet to begin with is for one specific use case: when the recipient wants to receive an exact amount.

With our modern view of ILP including micropayments and streaming payments, this seems like less of a core use case. But even if we still think this is an important use case, perhaps there is a better way to provide this functionality?

I believe there is. That is, by giving the recipient a way to reduce the amount received. The recipient is the party that cares about receiving an exact amount.

My suggestion to address this use case is that ledgers which serve merchants who have this requirement would allow merchants to pass back an amount together with the fulfillment. This amount must be lower or equal to the transfer amount and will be used to lower the actual amount transferred. The connector may also decide to fulfill their incoming payment with a lowered amount if they have a similar concern.

(One example where the connector would "pass on the savings" is a 1:1 connector that wants to keep the incoming and outgoing amounts exactly the same.)

@emschwartz suggested a slightly different method: The receiver could reject the payment with an "amount too high" error. This error could be addressed by the connector by re-sending the transfer with a lower amount, or it could pass the error on to the next connector or even all the way to the sender.

Either way is fine and either way would solve the "exact amount" use case in a much simpler way than the current solution.

Benefits of this change

• Simplify connector behavior

  The whole forward vs deliver goes away entirely. This is a currently a complex and awkward part of ILP.

• Allow more extensibility

  Forward vs deliver makes it harder to extend payment chains - once a connector in the chain thinks it is delivering (instead of forwarding) I can no longer have further currency exchanges later in the chain.

• Further simplify ILP packet format

Downsides of this change

• Changes ILP packet format

  The one thing we thought had finally stabilized. We could keep the amount field in the ILP packet and just permanently set it to zero or 0xffffffffffffffff.

• Requires solving the "exact amount" use case in another way

[justmoon]

If changing the ILP packet format is a major no-go, we could also keep it and just change who looks at it. I.e. connectors simply ignore it and it becomes a pure end-to-end field, like data. That would mean that the transport layer protocol defines how to interpret the amount field.

[justmoon]

Sidenote that this (or #311) would also enable an echo/ping mechanism to be purely end-to-end.

[emschwartz]

🤦‍♂️. I really thought the ILP packet format was finished. But I was never a fan of the whole forwarding vs delivery concept anyway... (and we have always said it's only done when there's nothing left to take away)

I think solving the exact amount issue on the connector level is much better than doing it on the ledger layer. Doing it on the ledger level adds another feature we'd expect from ledger plugins that could be pretty complicated to implement. In contrast, connectors would have a strong incentive to retry payments that are rejected because the amount is too high, because they would be able to pocket the difference.

If the packet would just be account and data, I'd wonder if we need that packet format at all, but I'll need to think about this more.

[sharafian]

If we remove the amount from the ILP packet, then we're really not serving the "receive an exact amount" use case at all anymore.

First of all, because both of the proposed solutions only deal with the situation where too much makes it to the end, the sender must always err on the side of too much. If the ILP packet had an amount, the connector could shift their rate minutely in order to make sure the correct amount gets to the receiver, even if rates have slipped. Without the amount in the ILP packet, the connector can no longer help in that way. The result is that the sender must always send more than they think is required in order to make the payment work. That's just like the current system, which we're trying to fix.

Adding a "partial fulfill" can be helpful to the receiver, but unless everybody in the payment chain implements it, the sender will still be paying out their amount along with the buffer they added to make sure it goes through. This means that fees will be higher for senders, potentially creating a worse experience for them. Implementing partial fulfill may also be difficult for some systems.

Rejecting an amount that's too large is also unhelpful to the sender. The receiver can reject and request a smaller amount, but it must go all the way back to the sender for the savings to make it there. Imagine a payment with one volatile hop. It may only be fluctuating by a tiny amount, but the payment must be completely reset every time. If we're dealing with any ledger fees, that's likely to be unacceptable to the parties. It also requires connectors to implement logic to deal with the receiver's desired amount in order for the payment to complete, removing some of the end-to-end properties.

I would advocate for special behavior in the connector where an ILP packet amount of 0 makes the connector ignore the amount. This fills all of our use cases, while leaving all current functionality untouched.

[emschwartz]

I think the idea is that if the sender streams the payment and chunks it up into little bits, they would send the right amount by stopping the stream when enough money has arrived. It does beg the question though of how big a bet we should put on streaming payments being the way to go. I think @justmoon would argue something along the lines of "go big or go home" (though in this case it's technically go small...).

[sharafian]

Well if we add the "ignore ILP amount when it's 0" logic, we would be enabling streaming in that way without deprecating the use cases that work right now

[emschwartz]

(FYI I picked the max value for #311 because you might actually have a reason to deliver 0, whereas it seems much less likely that you would try to deliver 18446744073709551615 units. If connectors do any type of delivery it makes more sense to me that they would round the number down but that a max value would mean something like "as much as you can get there")

[sharafian]

I think my main issue with changing things like the amount and quoting to be end to end is that they hack interledger messaging onto interledger payments. That makes it more general and therefore easier to add to, but also makes it harder to optimize. Things like caching quotes, filtering traffic, preemptively rejecting transfers that I know will fail, and delivering the exact amount that is needed are no longer possible if ILP doesn't differentiate between quotes, payments, and payments without a fixed destination amount. My feeling is that makes it less suited to the use case of making payments really well

[justmoon]

@sharafian

If the ILP packet had an amount, the connector could shift their rate minutely in order to make sure the correct amount gets to the receiver

That's not how it works. In practice, each connector forwards what it can according to its own rate and then the last connector checks if what it would be delivering according to its own rate is more than the amount. If yes, it delivers the amount in the packet, otherwise, it returns an error. The sender has to make sure to send enough money on the input.

All three solutions (current, and both options in the OP) result in the exact same behavior financially. It just becomes a lot easier to implement connectors and it makes it a lot easier to extend the payment chain outwards (for example if I want to create my own currency despite only having a regular bank account.)

You might counter that in the future, connectors might be smarter than today and do the behavior you describe where they fudge their rate in your favor. But I don't see how that would ever make sense to do. If they quote a worse rate than they are actually willing to do, then senders will just send a little bit less, hoping that the connectors' fudge factor will still make the payment succeed. So all you've accomplished is that you've moved the extra slippage into the quote rather than on top of the quote. But you haven't made things any cheaper or more efficient for the sender.

Not to be tedious, but I also want to reemphasize the benefits of this change. The forward vs delivery distinction is actually awful. (Yes, I advocated for it. This humble pie doesn't taste great.) It basically makes it so that there is a connector that cuts the payment chain in half. Before the first delivering connector, you're in amazing Interledger wonderland where anything is possible. After the first delivering connector, the currency is locked in. That's not true internetworking. In true internetworking, there shouldn't be any special hops in the chain, it should be infinitely extensible.

the sender will still be paying out their amount along with the buffer they added to make sure it goes through.

The sender always pays the buffer - by definition. They have to choose the sending amount before the connectors decide how much to forward. Rates could change in the meantime, therefore senders have to add a buffer. Whether you include a recommended buffer in the quote or add it on top makes no difference.

The fact that the sender pays for the buffer is actually a strong argument why the sender should be the one to decide the size of the buffer. Only they know what trade-off they want to make between success chance and optimum rate. (Higher slippage meaning higher success chance.)

I would advocate for special behavior in the connector where an ILP packet amount of 0 makes the connector ignore the amount. This fills all of our use cases, while leaving all current functionality untouched.

Interestingly, connectors already generally ignore the amount. Only one connector uses the amount: the first connector that is delivering instead of forwarding. So perhaps a better way of looking at this is that it isn't a change to the ILP packet format, but rather we're just phasing out delivering connectors.

I picked the max value for #311 because you might actually have a reason to deliver 0, whereas it seems much less likely that you would try to deliver 18446744073709551615 units.

Totally agree with that reasoning, my main concern with 18446744073709551615 is that it's hard to read or write. 18213129912919410 is just a large payment, 18446744073709551615 is the special magical payment... very easy to misread. We could fix that by special-casing the JSON format and writing this as "max" instead of "18446744073709551615".

But since I don't think there is any benefit in keeping the "delivering connector" behavior, I'd advocate we just remove it. If we remove that behavior, then we can put "0" into the amount field for quoting, resulting in a simpler, nicer-looking packet. (And transport layer protocols can still have "0" to mean "fulfill with a zero amount" if the data indicates it's not an ILQP packet.)

---

(As an aside: I don't think I like this idea very much, but we could use the same base64 trickery in the data field, where hex 0x20b40f encodes to ILQP in base64. But ultimately, it's probably more confusing than helpful to optimize for base64 too much.)

[michielbdejong]

I think the only reason to have an ILP packet, rather than just separate destination address, end-to-end data, etc, is that it allows the recipient to see if they are fulfilling the amount which the sender intended to send. A packet without an amount cannot fulfill that role. The sender should always care about how much money reaches the recipient, otherwise intermediate connectors can charge higher fees. So my worry here is that if there are situations where the receiver doesn't check the last transfer amount against the packet amount, that would lead to higher connector fees.

Same concern basically as my objections against remote quoting and end-to-end quoting; we should build a price-aware network, so that the end-to-end costs of using the network are driven down, and connectors need to compete for business by charging lower transaction fees.

[emschwartz]

allows the recipient to see if they are fulfilling the amount which the sender intended to send

I agree that it is important for the receiver to know how much they were supposed to get and only fulfill payments where the amount is at least that much.

@michielbdejong I think you misunderstand this proposal slightly.

The two main options proposed here are:

• take the amount out of the packet and make purely a concern of the transport / end-to-end layer (in which case it would make sense to include this inside the end-to-end data field and possibly encrypt it. You would definitely want the transport layer to specify this so connectors don't charge higher fees than they should)
• keep the amount in the packet but specify that connectors, not receivers should ignore it (the current title is a bit confusing in this regard). The main reason to do this is just to avoid making a breaking change to the packet format, and the possibility that there may be use cases where it makes sense to have it in there

[sharafian]

That's not how it works. In practice, each connector forwards what it can according to its own rate and then the last connector checks if what it would be delivering according to its own rate is more than the amount. If yes, it delivers the amount in the packet, otherwise, it returns an error. The sender has to make sure to send enough money on the input.

I know how the forwarding/delivery distinction works in the current code, but the precise way it's implemented is an implementation detail. The important thing is that the amount in the ILP packet reaches the end; right now we put the responsibility for that onto the last connector. If we wanted to, we could make a connector that double-checks their rates and nudges the amount a little bit even if they aren't the last hop.

The real problems that forwarding and delivery exist to solve are:

1. Determining whether the address in the ILP packet is an account on a ledger I'm connected to.
2. Doing one's best to get the exact amount in the ILP packet to the destination, to counter changes in rates.

Forwarding and delivery exist in our implementation because those two problems are solved by the last connector. (or rather, the first connector who has a plugin with a ledger prefix which is a prefix of the destination account).

The sender always pays the buffer - by definition. They have to choose the sending amount before the connectors decide how much to forward. Rates could change in the meantime, therefore senders have to add a buffer. Whether you include a recommended buffer in the quote or add it on top makes no difference.

Ideally, the connectors should know how much buffer to pay (like in the current implementation), because they're the ones setting the rates. If the connectors are giving quotes that immediately fail, they're not very good connectors. If they have an amount in the ILP packet, they can improve the chances of the payment succeeding by taking a little bit off their margins.

Generally, I have a few questions I think we might want to think about as we discuss these issues:

1. What are the responsibilities of the Interledger layer?
2. Does an address have a ledger prefix and an account?
3. Can a ledger prefix be the prefix of another ledger prefix?
4. Must every ledger support rejection? (as of right now, it's technically optional)
5. What optimizations is it safe for a connector to make?
6. How often do we expect rates to change?
7. On what order to we expect connector fees to be?

My answers are currently:

1. The Interledger layer gets a precise amount of money to a receiving entity, because this is easier to build on top of than building precise payments on imprecise ones.
2. Prefix and account are distinct, because the LPI makes that distinction.
3. Ledger prefixes cannot overlap, because we have no way to determine which overlapping prefix is the right plugin.
4. We should make 'reject' required by ledgers, but it should only be used when the entire payment is going to be put into a failed state.
5. The connector should be able to cache quotes or fetch them from a backend that doesn't propagate the quote all the way down the path.
6. I believe rates will change often, but not by so much that the jitter can be covered by slippage
7. Connector fees should be on the same order of magnitude as the jitter in rate changes.

I'm pretty sure that a lot of us would have different answers, though, which I think is why we're disagreeing on these ILP/ILQP changes.

[emschwartz]

I know how the forwarding/delivery distinction works in the current code, but the precise way it's implemented is an implementation detail.

Not really because we need to set expectations for what behavior people building on top of ILP can rely upon. The fact that this behavior is not clearly stated in RFC-3 right now is a problem.

Ideally, the connectors should know how much buffer to pay (like in the current implementation), because they're the ones setting the rates.

If all they do is pass on payments at their normal rate, they don't have to think about whether they're adding a buffer at all. They just consistently apply the rate, and the sender figures out how much they have to send.

If they have an amount in the ILP packet, they can improve the chances of the payment succeeding by taking a little bit off their margins.

If I knew you were doing this it would be very easy to take advantage of to always get that lower rate. If you're willing to take that lower rate, why not just offer it all the time?

1. The Interledger layer gets a precise amount of money to a receiving entity, because this is easier to build on top of than building precise payments on imprecise ones.

What do you think the normal Maximum Transfer Size will be for Interledger? Do you think it will be on the order of $1, $10, $100, $1000, or $10k? I would argue that if it's anywhere between $1-100 delivering the precise amount for a particular payment. If you're doing anything where the payment is very small I don't think either you or the merchant would care about you paying an extra cent. If you're paying for anything larger, you need to chunk up your payment anyway, in which case you would prefer for the maximum possible amount to arrive for each chunk, rather than caring whether it's exactly the amount you thought it would be.

I would argue that the end-to-end quoting thing demonstrates that imprecise payments, as you called them, are easier to build on top of. Also, why don't you like the alternative solutions for this proposed in the first comment?

2. Prefix and account are distinct, because the LPI makes that distinction.

I think it's different for a ledger plugin to have logic to determine the next address it should send to than for a connector to decide that there are no connectors beyond it that have non-1:1 exchange rates (even if there are additional segments in the address). In general, I don't think it's a very meaningful distinction. If I have a bunch of listeners on my "account" that each keep a ledger of payments they've sent or received and maybe even have a concept of a balance, why aren't those also considered real ledgers?

3. Ledger prefixes cannot overlap, because we have no way to determine which overlapping prefix is the right plugin.

Always take the longest prefix that matches.

4. We should make 'reject' required by ledgers, but it should only be used when the entire payment is going to be put into a failed state.

I agree that it should be required. What do you mean by "it should only be used when the entire payment is going to be put into a failed state"?

5. The connector should be able to cache quotes or fetch them from a backend that doesn't propagate the quote all the way down the path.

Why is that an important design consideration? Why not just make a connector that is exceptionally good at one thing and one thing only: forwarding payments?

6. I believe rates will change often, but not by so much that the jitter can be covered by slippage

Depends what assets you're talking about, how deep the markets are, how many speculative attacks or bubbles there are, etc. I don't think we can make too many assumptions either way but I don't see how this changes the design much.

7. Connector fees should be on the same order of magnitude as the jitter in rate changes.

Why does that matter?

[emschwartz]

I just realized that the fact that the ilp client module currently defaults to rejecting all transfers that do not exactly match the amount in the ILP packet makes it incompatible with @michielbdejong's alternate connector implementation because it doesn't implement delivery. We should try to avoid this type of subtle gotcha in the protocol.

[michielbdejong]

Ah ok, moving packet.amount into packet.data would just be moving things around, that's harmless.

The way my alternate connector determines the onward amount is by applying the curve to the amount from the packet, that way the connector always tries to take maximum profit. For a local ledger that curve would be 1:1, so then the transfer amount would be set equal to the packet amount, so I think the ilp module would react fine to that.

[sharafian]

If all they do is pass on payments at their normal rate, they don't have to think about whether they're adding a buffer at all. They just consistently apply the rate, and the sender figures out how much they have to send.

My point is just that the connector is better equipped to figure out the buffer, because they have a more thorough knowledge of the rates and how volatile they are.

If I knew you were doing this it would be very easy to take advantage of to always get that lower rate. If you're willing to take that lower rate, why not just offer it all the time?

A connector may only subsidize rates if they move in a certain way, or if some other unlikely event occurs. A sender could try to game that but it usually won't work.

If you're doing anything where the payment is very small I don't think either you or the merchant would care about you paying an extra cent. If you're paying for anything larger, you need to chunk up your payment anyway, in which case you would prefer for the maximum possible amount to arrive for each chunk, rather than caring whether it's exactly the amount you thought it would be.

Maybe it's something you can tolerate, but imprecise amounts are at best a nuisance. For a merchant who has to pay taxes or keep receipts, it's a headache to deal with payments that come out unpredictably. When you're at the grocery store and you pay $2.99, they still give you a penny in change, because they have to.

As for grouping a larger payment into several small payments, it's easier to deal with if you know the size of each chunk. If I'm sending $1000 in 10 $100 chunks, it's not like an extra 10 cents on each payment is going to make it go faster. It's just going to make the received amount unpredictable. You can engineer a protocol to communicate back and forth about how much actually got there but that kind of reconciliation is a huge pain.

I would argue that the end-to-end quoting thing demonstrates that imprecise payments, as you called them, are easier to build on top of. Also, why don't you like the alternative solutions for this proposed in the first comment?

The end-to-end quoting demonstrates how you could quote on top of imprecise payments, but not how you would implement precise payments. The two proposed solutions here address that, but I still think they have problems.

I don't think fulfilling for less is a good idea, because it's a complicated feature to add to the ledger. We've talked about ledgers that support non-standard features like this, but so far we've never needed any of them to serve an important use case. It also loses the intuitive property of all or nothing, which is confusing to explain.

Sending a rejection message to request less money be sent is difficult too. Currently, a single rejection means that the ILP payment has failed, and it is propagated back through the chain. Having a rejection that travels part way back and then travels forward again makes the payment flow more complicated and makes it easier for bugs to occur. We're introducing more steps to the commit/execute flow to solve something we could solve with an addressing convention.

Furthermore, no ILP errors are currently treated programmatically outside of their status code type. Having some human readable errors and some machine readable ones will make implementation of new connectors more difficult.

What do you mean by "it should only be used when the entire payment is going to be put into a failed state"?

Basically I just mean we shouldn't partially reject and then turn around and start preparing again.

Connector fees should be on the same order of magnitude as the jitter in rate changes.
Why does that matter?

If we expect connector fees are at least as large as rate changes, a connector can subsidize small rate changes when they occur without taking a loss in order to make sure that payments succeed.

The connector should be able to cache quotes or fetch them from a backend that doesn't propagate the quote all the way down the path.
Why is that an important design consideration? Why not just make a connector that is exceptionally good at one thing and one thing only: forwarding payments?

The connector is doing two things though, even if we overload them into a single operation. If forwarding a payment is being used to forward payments and send quotes, they can make fewer assumptions about how payments should behave. And if there's a way for quotes to not be forwarded across the whole path, that's a huge efficiency gain for us.

The whole "one thing and one thing only" philosophy is about making many simple tools, rather than making one general tool that does different things.

[adrianhopebailie]

It feels like this discussion is occurring at a fork in the road where we go from defining a protocol that could (and demonstrably does) work today to one that is designed for a future state we believe will emerge.

As someone who is trying to apply ILP to the payments use cases of today, I consider streaming payments an interesting experiment but not something I would remotely consider if it meant jeopardizing the ability to use ILP for the most common use cases we have today (i.e. fixed receive amount).

To that end I am strongly in favour of this being explored in a way that doesn't position it as something that must be implemented at the expense of what we already have, and I believe this is possible.

To summarize my counter-proposal in #313:

• We have to be able to differentiate between delivery vs forwarding otherwise we can't deliver fixed amounts. I see no way to solve this crucial use case without this logic implemented in all connectors.
• We can do this by making the differentiation between the ledger part and account part in an ILP address more explicit rather than forcing a quote to go end-to-end by pretending its a payment.
• The entity that should be responsible for differentiating between the ledger part and account part in an ILP Address is the receiver when they shares their address with anyone that wants to send them a payment.
• When a receiver is assigned an ILP address on a ledger (e.g. g.ledger.bob) they MUST always append a single segment to this address when sharing this with others as the destination for a payment (this may contain transaction specific information or be a random or constant value e.g. g.ledger.bob.~ or g.ledger.bob.invoice1234 or g.ledger.bob.8374568375683756).
• A connector MUST forward any payment or quote where the destination address has more than two segments added to the address of the ledger on which they are making the transfer or forwarding the quote.

NOTE: A connector could still cache quotes but they can only do this after forwarding at least one quote end-to-end. Connectors can now safely drop the last segments of any address when adding it as a routing prefix to their routing table. A connector could also aggregate routes further but this would be advanced connector functionality that we'd expect to only emerge as the network grows and connectors are able to make informed decisions about when to do this (probably based on a lot of historic data).

[michielbdejong]

we would like the connector to forward payments according to their offered rate

A rational connector would not voluntarily give up their high profit margins, the only way to force them to do that will be if we make connectors compete on the rate between source and destination amount.

Think of the source amount as what you pay for a T-shirt at H&M, and of the destination amount as the portion of that that gets forwarded to the garment workers in Bangladesh. Sure we "would like" the shop's "forwarding rate" to be as cheap as possible, but the only way to implement that is if the receiver puts their foot down and says "you didn't pay me enough, I'm not giving you the fulfillment until you do". Receivers should not produce the fulfillment unless the destination amount is at least as high as the packet amount. If the incoming amount is higher, they should pocket the difference. Connectors should do the same. They should keep a detailed view of how much they need to pay for which destination, and try to never pay too much. They should also keep a detailed view of how much they can charge for their forwarding service, and charge as much as possible. Competition between connectors will then bring the prices down.

The recipient is the party that cares about receiving an exact amount.

I would say they care about receiving a minimum amount in exchange for the fulfillment. Why would it be bad for a receiver if they receive too much? They can always throw the excess money away instead of giving it back, right?

giving the recipient a way to reduce the amount received.

I would say the opposite, if a connector charges 5% fee, the amount received by the recipient has been reduced, and that's a bad thing, not a good thing; the last thing I would want to do as a (rational) receiver is to say to that connector "Oh, but I feel your services is worth a charge of 7%, so here's some more money for you".

[justmoon]

Maybe it's something you can tolerate, but imprecise amounts are at best a nuisance. For a merchant who has to pay taxes or keep receipts, it's a headache to deal with payments that come out unpredictably.

There is no payment system that I'm aware of that delivers the exact gross amounts to merchants. Credit cards, PayPal, etc. all subtract fees from the paid amount. So you get the invoiced amount minus a couple of percent. Sure, the UI shows you the gross amount, but your actual balance changes by the net amount.

Note that from the merchant's perspective, they wouldn't be looking at raw ILP packets. That's like using tcpdump to look at your incoming HTTP requests.

What the merchant would see depends on the application layer protocol and the specific UI they are using, but I would imagine it would look something like:

Invoice # 123
Gross amount	$200.00
Cashback/Interchange	-$1.30
Fees	-$0.19
Net amount	$198.51

Gross amount is the amount on the invoice. Cashback/Interchange is the money that the merchant pays to the user (or their wallet) for using Interledger. (This is a requirement for adoption. Otherwise, users/wallets would never choose Interledger over cards which offer this.) Fees represents the 0.1% fee levied by the ecommerce platform being used, e.g. Shopify. In this example, any ILP overpayment is simply subtracted from the fees.

The merchant would book $200 sales income with a payment processing expense of $1.49.

To describe what's going on here more generally: On the application layer, we may very well care about showing a precise gross amount. But below that there are all sorts of little fees and costs. Senders may pay to quote and submit transfers. Receivers may pay a small fee to submit a fulfillment or rejection. (For example, on XRP Ledger you pay a small amount of XRP for EscrowFinish and EscrowCancel transactions.) So you already don't end up with the exact amount. I just don't think the connector delivery feature does anything useful.

If I'm sending $1000 in 10 $100 chunks, it's not like an extra 10 cents on each payment is going to make it go faster.

No, but it is going to make it cheaper. Note that with the old way, the sender and receiver have paid up to $1 extra to connectors.

To illustrate, here is the same example assuming a buffer of $0.10 per chunk and an actual average cost of $0.01 per chunk.

With exact delivery:

1st, 2nd, 3rd, ..., 10th payment
Sender sends $1000.10
Receiver receives $1000

With max delivery:

1st, 2nd, 3rd, ..., 9th payment
Sender sends $1000.10
Receiver receives $1000.09

10th payment
Sender sends $999.09
Receiver receives $999.08

Total cost with the old way: $1.00
Total cost with the new way: $0.10

Namely, with exact delivery, the buffer goes to the last connector as profit. With max delivery, the buffer goes to the recipient (and can be subtracted from future packets.)

In summary: We can't avoid having a buffer, because rates move. Given that we have a buffer, we need to figure out who gets the buffer. Our previous answer (forward-vs-delivery) sucks, because it arbitrarily rewards one player (last connector) for the total volatility across the path (which they had nothing to do with) at the expense of the users (sender and receiver.)

My point is just that the connector is better equipped to figure out the buffer, because they have a more thorough knowledge of the rates and how volatile they are.

1. The connector knows their local volatility, but they don't necessarily know anything about the volatility on other connectors. You don't want to add cumulative buffers, because the volatility across different assets may not be correlated or even inversely correlated. For example, consider the path USD -> XRP -> BTC -> JPY.

   USD -> XRP, XRP -> BTC and BTC -> JPY may all be very volatile. If each connector chooses a buffer, they would each add a large buffer. But you may not need a buffer that is 3x as large. USD -> XRP and XRP -> BTC will not be very correlated, so across those two pairs we may only need 1.4x the buffer of either of those pairs. USD -> XRP and BTC -> JPY may even be inversely correlated (because XRP-BTC and USD-JPY are correlated), so we may only need 1.1x one of the buffers to cover the volatility for both.

   The sender is in a good position to adjust the buffer, because they have a view of the whole path. In a streaming payment, I will quickly gain a sense of the path volatility. By the time I get to the 10th payment (from the previous example) I may have a very precise sense of the optimum buffer, because I have nine data points on the whole path volatility. Per the end-to-end principle, we should not try to have each connector keep track of the streaming payment and estimate volatility on behalf of the sender. Not to mention that connectors don't really have a way to communicate new information about changing volatility to the sender anyway (unless the sender quotes over and over.)

2. The connectors don't know the sender's preference which is the most significant factor influencing the buffer. If the sender doesn't need more than a random success rate, they may not want to put a buffer at all. If the sender really needs the payment to succeed no matter what, they may include a very large buffer.

I don't think fulfilling for less is a good idea, because it's a complicated feature to add to the ledger.

Yeah, I don't think it would be worth adding this feature just for the fixed delivery amount use case. I was merely pointing out that if it turns out that recipients care for some reason, they can use ledgers that support this feature.

Another solution is to use a client which hides the buffers and accumulates them until they reach $1 and then quietly sends them to the Red Cross. That way you can take your strange obsession with not being paid extra and turn it into a virtue. ;)

It also bears mentioning the impact on our primary use case: micropayments. For HTTP-iLP, we could credit the buffer to your token.

The connector is doing two things though, even if we overload them into a single operation. If forwarding a payment is being used to forward payments and send quotes

I think it's a bit wrong to call the new flow "quoting". What "ILQPv2" is doing is simply figuring out the cost of a payment, by making a payment. So quoting simply becomes unnecessary. Therefore I don't really see it as overloading, but eliminating quoting.

This might be more clear in the case of a streaming payment. Here you wouldn't typically reject the quote payment, but rather accept it (I'm assuming #314 here) and simply use it as the first of the streaming payments. You probably also wouldn't call it the "quote payment" but rather the first payment in the stream.

Actually, writing about it now, I don't see why you wouldn't always do that.

The flow would be something like:

A sender would like to pay a recipient.

1. Sender sends an initial payment with a small constant amount.
2. If the payment is (significantly) larger than the expected amount, the recipient rejects it with an error F08 Amount too large, returning the actual amount they received
3. If the payment is less than or equal to the correct size, the recipient fulfills it and returns the amount they received and the remaining amount to be paid.
4. While the remaining amount to be paid is greater than zero, the sender makes another payment that is min(remaining, max-packet-amount).

(One feature we may want to have is for the second and following payments is for the sender to include a "minimum delivered amount" - which would go in the ILP packet's amount field. The recipient would reject the packet if the amount is less than this which prevents an unexpected crash in the rate to eat up a whole packet. When getting such a rejection, the sender can re-evaluate whether they want to continue the stream at the new rate.)

And if there's a way for quotes to not be forwarded across the whole path, that's a huge efficiency gain for us.

Surely, not quoting at all is an even greater gain. :)

[michielbdejong]

I tried to explain the idea of Interledger as an open network of rational agents a bit better in #315, HTH.

[emschwartz]

In #316 @michielbdejong is convincing me that connectors attempting to deliver as close to the amount in the packet as possible is a somewhat inevitable result of the amount being in the packet as cleartext. Since I don't think the delivery behavior is desirable, I'm now leaning more towards the position that the amount should be taken out entirely (as much as I hate the idea of changing the ILP packet, because I was sure that was done).

In the same vein, I think we should make new versions of the Transport Layer protocols that don't send anything in the clear -- even which protocol or protocol version they correspond to. I've thought of various protocols that could ride on top of ILP where you'd have a reason to make the packet look like a PSK packet. If we end up with a bunch of use cases like that, we'll end up with every packet having the data start with PSK/1.0\n, in which case we might as well not have any start with that. Receivers will have different ILP addresses and will know whether a packet is meant for them from that and whether they can decode the data.

In a highly optimized ledger that might not be such an issue

There are so many reasons we're going to want payments to be fast. If XRP escrow is too slow, I think we might want to just bite the bullet, accept that everything will be payment channels and trustlines for now, and then work on making that experience as good as possible for end users.

eschew quoting entirely for a local ILP payment

This actually has the same problem you brought up here. In order to know that it's local you need to make assumptions about the ILP address. And it's worse because if it is local and you try to request a quote, it won't work because receivers don't implement ILQPv1. I realized this while trying to play with a simple BTP client and server and found I couldn't test out a full SPSP/PSK/ILP/ILQP/BTP implementation with just that setup because I also needed a connector that would implement ILQP in the middle. An advantage with end-to-end quoting is that you always have a sender and a receiver, but you don't always have a connector in between. With end-to-end quoting you don't need two different flows depending on whether you (think you) are on the same ledger.

I also think there's still a big messaging problem (from a marketing and adoption standpoint) with ILP not guaranteeing a precise amount to the receiver.

Quite possibly. Worth thinking about more.

Finally, our current system works, and we've been building applications on top of it. Changing it again is going to cause more breaking changes and incompatibilities than we expect. Having our protocols stable has been so useful; I would be sad to give that up again.

Agreed. 😞

(Though I'm still holding on to the idea that we'll have to stop changing things when there is nothing left to take out)

...And with that, I'm off (of the grid) for the next week. Looking forward to rejoining this discussion next Sunday!

[michielbdejong]

If the payment is (significantly) larger than the expected amount, the recipient rejects it with an error F08 Amount too large, returning the actual amount they received

That means the receiver is not modelled as a rational (greedy) agent, then? What would incentivise the receiver to be honest about receiving too much?

[emschwartz]

The sender is always going to judge your prices based on the (source) amount it will cost them. There is no difference between the receiver lying about how much they received and the receiver just asking for a higher amount in the first place (since the amount is in units that are meaningless to the sender anyway). What stops the receiver from raising their prices? Depends on the market for whatever product or service they're offering.

[adrianhopebailie]

The sender is always going to judge your prices based on the (source) amount it will cost them.

It's not quite as simple as that. There are many use cases where the receiver will use a known unit (like USD) so that the sender can compare costs without needing to initiate the payment process and get a quote for each potential supplier.

[emschwartz]

Just because something says it's denominated in USD, doesn't mean it will have the same exchange rate as another ledger that's supposedly denominated in USD. In a really wide open system, it's unwise to build things with the assumption that the asset type has meaning independent of the ledger it's on.

[emschwartz]

Does anyone have objections to special-casing an amount of 0 and keeping the behavior the same for other amounts? A zero amount would mean "just forward this according to your local rate". That way we can avoid changing the stack a whole bunch right now while leaving room for protocols to be built on top of ILP that don't want connectors to try to deliver a specific amount.

cc @justmoon @sharafian @michielbdejong @adrianhopebailie

[adrianhopebailie]

Does anyone have objections to special-casing an amount of 0 and keeping the behavior the same for other amounts?

Definitely in support of this! #313

[michielbdejong]

Does anyone have objections to special-casing an amount of 0

Yes, I'm very much against making the meaning of https://github.com/interledger/rfcs/blob/master/asn1/InterledgerPacket.asn#L30 ambiguous, see my counter proposal which introduces a new call type, so that within each call type, the meaning of a packet does not depend on its context.

[emschwartz]

I can see it both ways, but I have to ask how ambiguous is it really? With interledgerjs/ilp-connector#392 you would take a 0 amount in the packet to mean send an outgoing transfer with an amount of 0. Arguably you're going to need to special-case that anyway, so why not make the special case behavior be something useful rather than just throwing an error?

[michielbdejong]

(from your remark on #312 (comment)):

there's a good chance a lot of ledgers won't let you send a 0 amount.

Intermediate connectors should not care what the destination ledger does and does not support. They just look at the curve. Even if the curve had to be read in a special way for destination amount zero, that would not be a reason to mix up payments that do, and ones that don't have an amount specified. There's a fundamental difference between 0 and undefined.

[adrianhopebailie]

There's a fundamental difference between 0 and undefined

Not in the case of a payment.

I still think we are trying to design the protocol based on how we want connectors to be implemented, which is incorrect. We should design the protocol based on the outcomes we desire. Designing specific implementations are out of our scope.

When a connector receives an incoming transfer it must decide where to send an outgoing transfer, for how much, and with what expiry. The protocol should not dictate how they come to that decision, that logic will be part of how connectors compete in the market.

If the connector receives an ILP packet with an amount of 0 (or null or whatever we decide) that should simply mean the connector has one less piece of data to use in making its decision.

What the protocol needs to specify is whether or not an amount of 0 is actually allowed. In other words, is it okay for a connector to reject incoming transfers if the amount in the ilp packet is 0 or is that bad behaviour that peers of that connector should consider a bug and therefor start routing payments around the "faulty" connector.

[michielbdejong]

how ambiguous is it really?

If, in a network, all connectors implement Current Interledger, there is no ambiguity. If all connectors implement your '0 means best-effort' proposal, there's also no ambiguity.

Ambiguity arises if some connectors will treat a 0 destination amount as best-effort, just to have their best-effort payment end up as a tip to the next connector, then that becomes very unpractical. All connectors need to be in agreement about what a packet means. We should not overload the meaning of one packet type to have multiple meanings depending on their context.

[michielbdejong]

The protocol should not dictate how they come to that decision

That's orthogonal, this issue is just one of mapping semantics onto syntax: for best-effort payments, which we all agree is a new feature that didn't exist a few weeks ago, do we create a new packet type? (I think yes) Or do we make a breaking change that will cause a fork?

[adrianhopebailie]

do we make a breaking change that will cause a fork?

We have never prescribed what a payment of 0 means so I don't think this is breaking. Either way if we introduce this new feature all existing connector implementations will need to be updated to handle the "destination amount unknown" case.

Whether they get a packet with a o amount or a new type of packet they will all break today, although there is a chance that some will not with the 0 amount.

So, assuming that the breaking change is that we start sending packets without a destination amount (ignoring how we do that) we should consider which way of doing will be easiest to support in existing implementations. I think the 0 amount will be.

For ilp-connector I'm not sure there would even need to be a change, for your connector you'd need to add some logic that says, if the destination amount is unknown use the curve for delivery to the next connector.

[adrianhopebailie]

p.s. At this point I think we may have spent more energy discussing this than is needed to just implement it 😄

[michielbdejong]

Nobody has described any downsides to #323 yet, right? So if we just merge that, then we're done.

We have never prescribed what a payment of 0 means so I don't think this is breaking

not true, we have not mentioned each possible amount but we have prescribed how the amount should be interpreted. that includes amounts of 0, 1, 2, etc.

[michielbdejong]

existing connector implementations will need to be updated

only if they want to serve that traffic - we can leave that to market forces. also, tif we introduce a packet type 9, they don't need to be updated all at once (which is what i mean with 'fork')

[michielbdejong]

Whether they get a packet with a o amount or a new type of packet they will all break today, although there is a chance that some will not with the 0 amount.

afaik all connectors now support delivering a zero destination amount. it can be used for messaging. the new packet type will only be forwarded by connectors that understand it.

[adrianhopebailie]

Nobody has described any downsides to #323 yet, right? So if we just merge that, then we're done.

It adds new conformance criteria for implementations that we are not sure we need.

we have prescribed how the amount should be interpreted

Where have we done that? I don't think we have. We have described what the value in that field is but not how implementations must behave based on that. We leave it to "market forces" as you say to drive the business logic in a node and how that will determine the outgoing transfer amount.

afaik all connectors now support delivering a zero destination amount

So let's document their behaviour as part of the protocol and then decide if we still need #323

[michielbdejong]

@emschwartz and I reached a consensus that it's better to implement #323.

[emschwartz]

Resolved by #323