Building a TLV extension app - a field report

[jarret]

As a challenge, I (with some help from others) built https://onion.studio to explore how the createonion, sendonion and generally the TLV stuff can be used for building apps. Also, maybe help spread awareness of the associated concepts out there. The full code is all at: https://github.com/jarret/onionstudio

First, thanks to all of you in the C-Lightning project for building this stuff, exposing it to be used from the outside and performing the handful of public talks to explain it. Very cool stuff that I am excited to follow the future of.

Here are some of my thoughts and feedback about some of the bumpy parts I encountered in my arc of building. A lot of it is along the lines of "other app developers will likely have need for the same stuff I wrote, so shared plugin/platform code that makes these operations easier would be a benefit". This type of thing also might be premature to action on without additional feedback from others trying to build apps.

I hope this account is useful info to help refine going forward. I apologize in advance for verbosity:

The documentation for createonion refers to BOLT 4 for how to encode a payload. That was a rabbit hole for me to go down and I ended up having to implement the stuff in BOLT 1 in Python in order to build the TLV payloads myself.

This library is somewhat generalized such that it could be used by a different Python app and has four main modules - bigsize.py, tlv.py, namespace.py and hop_payload.py which build on each other (in that order).

For my app-specific extension TLVs, I extend a generated hop payload with my extension module.

When writing this, I was thinking about how C-Lightning has the invoice and decodepay commands which, in a way, go together as a pair for creating and decoding a BOLT 11. invoice also is able to prevent the creation of an incorrect BOLT 11 with the benefit of context. The net effect is that the application developer doesn't have to think much about the details of BOLT 11 encodings, so maybe that is similarly achievable for the BOLT 4 payload encodings.

SUGGESTION 1:
It might be good to provide an 'on rails' payload creator that can to all the TLV encoding/decoding according to spec with the main codebase such that encoders/decoders don't have to be written in N different languages. Also, it is imaginably straightforward to include optional parameters for extension TLVs to be appended after.

Since a) there is the hard cap of 1300 bytes for the payload + HMAC data in the onion packet and b) the data is all variably-length encoded for compactness and c) The quantity of hops to get to the destination (and back) can also be fluid, it makes it difficult to determine how much data is ultimately available in the packet for application extension data.

My solution for Onion Studio is a fudgy estimate starting point and a couple iteration of finding hops and attempting an encoding, and reducing the amount of payload pixels I encode until it fits under the 1300 byte limit.

Also, when playing with this stuff I hit the two crashing bugs: #3377 #3370 which were additional pain for trying to use the createonion command before I had a proper clue about what I was doing.

Overall, this was kind of gnarly to get done, but I now have some solution logic written that might be helpful to others.

SUGGESTION 2a:
Aside from fixing the crashes and giving nice error messages when you give wrong input, it might be good to provide a fuller-featured onion packet creator functionality that is more aware of the content and give you less rope to hang yourself with. Application-side fudgy estimates like mine are doomed to be less accurate and maintained than the internal code for creating onions doing validation, so it seems to make sense to provide a better service.

SUGGESTION 2b:
Perhaps in conjunction with SUGGESTION 1, the workflow for such an onion creator tool might have two phases. First, a phase for creating the routing instructions as desired since that is the most important thing. Second, the first phase might tell you how many 'spare' bytes are left for non-routing/extension concerns and then lets you fill them in.

Onion Studio's client sends the pixel data via a circular route and pays for the pixels by 'overpaying' the routing fee at that hop. The route creation uses getroute with a fuzzpercent setting of 0.0. It uses the fromid to compute the reverse route in the same way, which is can be identical to the outgoing route. It has to be deterministic simply because the algorithm need to hold the route relatively constant as it iterates on the onion construction to fit the pixel payload and appropriate payment using the rest of the space (and changing the payment amount also can influence the chosen route, changing the available space, changing the number of pixels in the payload, changing the payment, etc.).

This implementation trickiness resulted in me abandoning the desire to construct a 'good' circular route that would prefer the outgoing and returning route to be fuzzy and avoid node overlaps. The result is an implementation that is perhaps sub-par for the purpose of passing a maximally-discreet message.

The chosen circular scheme is also convenient for sending data without some out-of-band negotiation between the source in the destination. An alternative for a similar app might be the "Key Send" scheme to make it a unidirectional send of a payment + extension data. However, choosing between the two schemes for a particular style of app is a complex discussion that I don't have a clear formed opinion on.

However, I observe that the circular scheme might be useful for obscuring the real destination of a 'TLV RPC call' among the set of participants in the circular route.

Also, I observe that if there are schemes designed (I expect this to be inevitable, if not already happened) for 'TLV RPC calls' where there is a call out with a payload to a node, and a call back with a payload to the source, this might look exactly like a circular operation, even though it might be two different unidirectional sub-operations in implementation. Having both schemes around and supported might be a good platform for privacy for the variety of possible operations.

SUGGESTION 3:
A "get circular route" operation might be a good plugin/command to provide as a common utility. Also, given that the 'real' payment might be in the form of a forwarding fee somewhere along the route, it would be a nice feature if that amount could be optionally be selected as a criteria.

A noted fact of the extension TLVs is that they are delivered to their intended recipient before the preimage is revealed to trigger the associated payment. Onion Studio's design is such that the pixels don't draw until the payment is received.

On the application side, it has to catch the 'payload' field upon a call from the htlc_accepted hook notification and hold on to the data it until getting a forward_event notification. Also, the application needs to have a scheduled 'pruning' step to cull the stored payload data from forwarding HTLCs that never get paid.

SUGGESTION 4a:
Possibly consider a design for notifications to the plugin to give notification of the extension TLVs specifically along their lifecycle such that the application can get a notification upon 1) HTLC accepted 2) HTLC fulfilled or 3) HTLC expired. This way the extension TLV might not need to be held separately by the application

SUGGESTION 4b:
One (perhaps half-baked) thought is that there might be value in the protocol supporting encrypted extended TLVs that can only be decrypted with knowledge of the preimage. This would be such that the recipient doesn't know what the requested extension operation is until the payment is received. This would give sender the option to avoid revealing the content of request until the sender has definitively decided to go ahead with the payment/operation.

With Onion Studio, typically only ~250 encoded pixels fit in the onion packet. Unless they are tiny .png images, they have to be split up into many separate payments. For example, a 200x200 pixel image will require approximately 160 payments of 250 satoshis each totalling 40,000 satoshis to transmit all the pixels.

There is a challenge in estimating channel capacity for this kind of payment because it isn't quite "I wish to route 40k sats to the destination", it is "I wish to route many small payments totaling 40k sats to the destination which can go via different routes". Ideally, we would want to determine for the user whether it looks like that can realistically be done to completion before deciding to proceed. Right now, the approach of Onion Studio is to allow failure if there is a capacity exhaustion event midway, but gives the user the option to manually resume the transmission from the point it left off later. However, this might be frustrating user experience to have to babysit the operation and restart.

I imagine this is a similar problem for other data-publishing apps where it doesn't really deliver full value to only get a partial set of payments through to their destination.

SUGGESTION 5:
This probably needs a chunk of science done to figure out good algorithms for this, but I can perhaps spot a similarity between this and the routing considerations needed for AMP. In this specific concern, we want the full message transmission to be the concern for atomicity. It might make sense to include this problem for consideration in that discussion.

~

FIN

[cdecker]

Thanks for the very detailed feedback @jarret, I'll try to address things
inline, and spin out separate issues when appropriate 😉

As a challenge, I (with some help from others) built https://onion.studio to
explore how the createonion, sendonion and generally the TLV stuff can be
used for building apps. Also, maybe help spread awareness of the associated
concepts out there. The full code is all at:
https://github.com/jarret/onionstudio

First, thanks to all of you in the C-Lightning project for building this
stuff, exposing it to be used from the outside and performing the handful of
public talks to explain it. Very cool stuff that I am excited to follow the
future of.

Thanks, I'm really glad that the onion RPC was useful for you. Onion.studio is
a very interesting idea, and given the right tooling these kinds of
applications could go a long way.

The documentation for createonion refers to BOLT 4 for how to encode a
payload. That was a rabbit hole for me to go down and I ended up having to
implement the stuff in BOLT 1 in Python in order to build the TLV payloads
myself.

This library is
somewhat generalized such that it could be used by a different Python app and
has four main modules - bigsize.py, tlv.py, namespace.py and
hop_payload.py which build on each other (in that order).

For my app-specific extension TLVs, I extend a generated hop payload with my
extension module.

When writing this, I was thinking about how C-Lightning has the invoice and
decodepay commands which, in a way, go together as a pair for creating and
decoding a BOLT 11. invoice also is able to prevent the creation of an
incorrect BOLT 11 with the benefit of context. The net effect is that the
application developer doesn't have to think much about the details of BOLT 11
encodings, so maybe that is similarly achievable for the BOLT 4 payload
encodings.

SUGGESTION 1: It might be good to provide an 'on rails' payload creator that
can to all the TLV encoding/decoding according to spec with the main codebase
such that encoders/decoders don't have to be written in N different
languages. Also, it is imaginably straightforward to include optional
parameters for extension TLVs to be appended after.

I finally sat down myself and published my code (#3423) for manipulating
varints and onion payloads, maybe we can merge the two to become a really
pythonic way of handling them?

I also think that examples are a good showcase on how things can be used, so
I'm happy you published the code for onion.studio :-)

Since a) there is the hard cap of 1300 bytes for the payload + HMAC data in
the onion packet and b) the data is all variably-length encoded for
compactness and c) The quantity of hops to get to the destination (and back)
can also be fluid, it makes it difficult to determine how much data is
ultimately available in the packet for application extension data.

My solution for Onion Studio is a fudgy
estimate
starting point and a couple iteration of finding hops and attempting an
encoding, and reducing the amount of payload pixels I encode until it fits
under the 1300 byte limit.

Also, when playing with this stuff I hit the two crashing bugs:
#3377
#3370 which were additional
pain for trying to use the createonion command before I had a proper clue
about what I was doing.

Overall, this was kind of gnarly to get done, but I now have some solution
logic written that might be helpful to others.

SUGGESTION 2a: Aside from fixing the crashes and giving nice error messages
when you give wrong input, it might be good to provide a fuller-featured onion
packet creator functionality that is more aware of the content and give you
less rope to hang yourself with. Application-side fudgy estimates like mine
are doomed to be less accurate and maintained than the internal code for
creating onions doing validation, so it seems to make sense to provide a
better service.

SUGGESTION 2b: Perhaps in conjunction with SUGGESTION 1, the workflow for such
an onion creator tool might have two phases. First, a phase for creating the
routing instructions as desired since that is the most important
thing. Second, the first phase might tell you how many 'spare' bytes are left
for non-routing/extension concerns and then lets you fill them in.

I'm still trying to reproduce #3370, which seems to get stuck when the next
hop is the local node, but #3377 has been fixed, and will now just return an
error if the payload is too long. The packet creator is a nice idea, and the
estimation should not be too difficult, we just need to re-implement the onion
serialization (skipping the encryption steps) to get the serialized size. I'll
sit down and see what I can come up with.

As you note the maximum payload is dependent on the route length and the type
of payload we deliver at each of the hops, so we can't give a limit a priori,
but we can definitely give exact sizes once the route is fixed. Maybe building
a higher level (object-based) interface on top of the raw RPC API might be
interesting to have a Route object that can be transformed into an Onion
object with appropriate fields to call createonion in the background.

Onion Studio's client sends the pixel data via a circular route and pays for
the pixels by 'overpaying' the routing fee at that hop. The route creation
uses getroute with a fuzzpercent setting of 0.0. It uses the fromid to
compute the reverse route in the same way, which is can be identical to the
outgoing route. It has to be deterministic simply because the algorithm need
to hold the route relatively constant as it iterates on the onion construction
to fit the pixel payload and appropriate payment using the rest of the space
(and changing the payment amount also can influence the chosen route, changing
the available space, changing the number of pixels in the payload, changing
the payment, etc.).

I don't quite follow why a circular route was needed to do this? A one-shot
payment with the appropriate payload, and an associated htlc_accepted hook,
could have implemented the keysend protocol, which communicates the
payment_key to the recipient and thus given it the ability to terminate the
payment, without needing the "drop fee here" trick :-)

FWIW I have a keysend implementation as part of the noise plugin,
which you might want to look into. It makes routing a lot simpler, saving
space in the onion for moar pixels 🚀

Also, I observe that if there are schemes designed (I expect this to be
inevitable, if not already happened) for 'TLV RPC calls' where there is a call
out with a payload to a node, and a call back with a payload to the source,
this might look exactly like a circular operation, even though it might be two
different unidirectional sub-operations in implementation. Having both schemes
around and supported might be a good platform for privacy for the variety of
possible operations.

Interesting thought. I don't quite follow how you'd add the response of the
TLV RPC call to the onion that is already addressed to back you. I think
another solution might be to use something similar to the noise
plugin to deliver the RPC call, alongside a signature or a pubkey of
the sender. The recipient then learns the identity of the sender (from the
pubkey or through pubkey recovery from the signature) and can then send the
result back to the sender. By not disclosing the payment_key in the call the
sender can defer payment for the call until it receives the response: the
recipient would simply reuse the same payment_hash (not secure since the
caller can still just cancel, but the same is true for circular payments).

SUGGESTION 3: A "get circular route" operation might be a good plugin/command
to provide as a common utility. Also, given that the 'real' payment might be
in the form of a forwarding fee somewhere along the route, it would be a nice
feature if that amount could be optionally be selected as a criteria.

Definitely agree with this one, I think we can build some very idiomatic
abstractions on top of the raw RPC, and we're just scratching the surface
here. I have been playing with circular payments a bit lately and we might
subsume some of the common patterns into a pythonic library built on top of
the raw RPC.

A noted fact of the extension TLVs is that they are delivered to their
intended recipient before the preimage is revealed to trigger the associated
payment. Onion Studio's design is such that the pixels don't draw until the
payment is received.

On the application side, it has to catch the 'payload' field upon a call from
the htlc_accepted hook notification and hold on to the data it until getting
a forward_event notification. Also, the application needs to have a
scheduled 'pruning' step to cull the stored payload data from forwarding HTLCs
that never get paid.

I think this use-case would have been better served using the keysend
protocol, allowing the plugin to immediately decide whether to draw or not
based on whether it can terminate the payment or it has to fail it. This would
have removed the caching of actions and the chances of failing even after
having cached the actions.

SUGGESTION 4a: Possibly consider a design for notifications to the plugin to
give notification of the extension TLVs specifically along their lifecycle
such that the application can get a notification upon 1) HTLC accepted 2) HTLC
fulfilled or 3) HTLC expired. This way the extension TLV might not need to be
held separately by the application

I think we have a couple of people asking for more fine-grained HTLC
notifications, so that might indeed be implemented some time soon. See
tracking issue #3424. The major question is what kind of payload should these
have? I don't think that passing in the onion upon each call would be
sensible: it's 2.6 kB of hex encoded binary data that we'd pass in 2 times,
and for forwards we'd add it 4 times in total (2 on the incoming side, and 2
on the outgoing side). I'd say we can have the onion in the htlc_added case,
and the plugin can cache the onion, being certain to get notified once it is
no longer of interest and can be cleared. Does that sound like a good
middleground?

SUGGESTION 4b: One (perhaps half-baked) thought is that there might be value
in the protocol supporting encrypted extended TLVs that can only be decrypted
with knowledge of the preimage. This would be such that the recipient doesn't
know what the requested extension operation is until the payment is
received. This would give sender the option to avoid revealing the content of
request until the sender has definitively decided to go ahead with the
payment/operation.

This should already be possible with the current protocol, just encrypt a
field in the payload using the preimage on the sender side, and then wait for
the HTLC to be settled.

With Onion Studio, typically only ~250 encoded pixels fit in the onion
packet. Unless they are tiny .png images, they have to be split up into many
separate payments. For example, a 200x200 pixel image will require
approximately 160 payments of 250 satoshis each totalling 40,000 satoshis to
transmit all the pixels.

There is a challenge in estimating channel capacity for this kind of payment
because it isn't quite "I wish to route 40k sats to the destination", it is "I
wish to route many small payments totaling 40k sats to the destination which
can go via different routes". Ideally, we would want to determine for the user
whether it looks like that can realistically be done to completion before
deciding to proceed. Right now, the approach of Onion Studio is to allow
failure if there is a capacity exhaustion event midway, but gives the user the
option to manually resume the transmission from the point it left off
later. However, this might be frustrating user experience to have to babysit
the operation and restart.

I have been talking with some other plugin devs about creating a Payment class
in the python library that can be inherited/customize to add custom behavior,
but implements the generic functionality to find alternative routes and retry
when a recoverable error is returned. This class could then be customized to
pull pixels from a queue of pixels to draw, packing as many of them into a
single payment as will fit, retrying if failed.

I imagine this is a similar problem for other data-publishing apps where it
doesn't really deliver full value to only get a partial set of payments
through to their destination.

SUGGESTION 5: This probably needs a chunk of science done to figure out good
algorithms for this, but I can perhaps spot a similarity between this and the
routing considerations needed for AMP. In this specific concern, we want the
full message transmission to be the concern for atomicity. It might make sense
to include this problem for consideration in that discussion.

I think AMP is likely not what you want here, since you want pixels and
payments to coincide. Your problem is pretty orthogonal to what AMP tries to
achieve, and I'd think that your amounts are small enough not to require
splitting into parts. A more standardized fail-retry-loop is more likely what
you need here :-)

[cdecker]

I'll leave this issue open for a couple of days until we've addressed all the details and/or spun them out into their own issues. Does that sound ok to you @jarret ?

[jarret]

Does that sound ok to you @jarret ?

Yep, sounds good. Thanks for the detailed set of thoughts. I am still processing and might need a few days to organize some thoughts back.

[jarret]

Onion.studio is a very interesting idea, and given the right tooling these kinds of applications could go a long way.

Thanks! Just from discussing and spitballing some ideas in various channels over the last few days, it is great to see that the ideas are starting to flow out there. I am feeling that the new-style tightly-coupled "payment + instruction" mode of interaction with a remote node kind of throws a wrench into some of the initial wave of thought for how LN UIs might turn out:

type words, add to cart, request invoice, pay invoice becomes ---> key press sends payment along with key

I finally sat down myself and published my code (#3423) for manipulating
varints and onion payloads, maybe we can merge the two to become a really
pythonic way of handling them?
I also think that examples are a good showcase on how things can be used, so
I'm happy you published the code for onion.studio :-)

I read through the code for noise and it was great to absorb. I learned a number of things and it is a good frame for related discussions. So likewise, thanks for publishing. :-)

I would love to continue work on collaborating on infrastructure and the larger LN and C-Lightning project in particular, but unfortunately, I can't commit to being responsive and reliable and don't want to slow anyone down. The progress and speed of this project is amazing and I simply don't have the spare cycles to keep pace in the way that I would want to. I am subscribed and reading most of what goes by here, and am glad to chip in as I can, though.

I'm still trying to reproduce #3370, which seems to get stuck when the next
hop is the local node, but #3377 has been fixed, and will now just return an
error if the payload is too long. The packet creator is a nice idea, and the
estimation should not be too difficult, we just need to re-implement the onion
serialization (skipping the encryption steps) to get the serialized size. I'll
sit down and see what I can come up with.

As you note the maximum payload is dependent on the route length and the type
of payload we deliver at each of the hops, so we can't give a limit a priori,
but we can definitely give exact sizes once the route is fixed. Maybe building
a higher level (object-based) interface on top of the raw RPC API might be
interesting to have a Route object that can be transformed into an Onion
object with appropriate fields to call createonion in the background.

All makes sense.

One additional thought from some of the other loose discussions around programming more complicated operations - There might be the need for programming a 'compound' route. Where Alice sends an onion to Bob, gives Bob a piece of data that instructs Bob to create an onion to Carol for Carol to likewise create an onion to Alice. Alice might release a preimage that triggers the set of pending payments in one of several pre-programmed ways that she decided in a late-binding way (perhaps contingent on what she hears back from Carol).

In a scheme like that, Alice would have to estimate 3 different routes with different data requirements before deciding to send. Thinking along those lines, my suggestions for something 'nice' to provide in a route creator/estimator might still be premature. It might be better to understand more concrete examples of cool ideas first.

I don't quite follow why a circular route was needed to do this? A one-shot
payment with the appropriate payload, and an associated htlc_accepted hook,
could have implemented the keysend protocol, which communicates the
payment_key to the recipient and thus given it the ability to terminate the
payment, without needing the "drop fee here" trick :-)

I'll be perfectly honest, I didn't connect that LND's KeySend was possible in the current C-Lightning release via response to the htlc_accepted hook until I read it in the noise code the other day. My mind was still pegged to a narrower understanding. I guess this is how I learn. :)

In my local Bitcoin meetup group, I have been doing my share of discussing and teaching people on the concepts of Tor and onion routing in general with the notions of a) circular routing and b) overpaying hops. These have value in creating an aonyminity set of several potential nodes that could be the 'real' destination in order to confuse any potential packet snoop attack. Also, the idea of disguising an important message/payment in an early hop as a more trivial-looking payment that manifests in the final hop.

Examining the new --data and --keysend options in the LND implementation, it looks like the command set only currently allows you to put a payload on the final hop. That strikes me as, on one hand, very pragmatic for getting started with the first wave of message-sending apps, but, on the other hand, is not quite satisfying the same programming power that has captured my interest.

I guess I am still in the anti-pragmatic mode of playing about trying to understand the larger set of potentials. At least for now, circular payments have a bit more to talk about. Also, I can report that as a result of whiteboarding the explanation circular payments for Onion Studio, I have also caused a few "ahah" moments for people understanding channel rebalances and how the sendinvoiceless.py plugin works.

Interesting thought. I don't quite follow how you'd add the response of the
TLV RPC call to the onion that is already addressed to back you. I think
another solution might be to use something similar to the noise
plugin to deliver the RPC call, alongside a signature or a pubkey of
the sender. The recipient then learns the identity of the sender (from the
pubkey or through pubkey recovery from the signature) and can then send the
result back to the sender. By not disclosing the payment_key in the call the
sender can defer payment for the call until it receives the response: the
recipient would simply reuse the same payment_hash (not secure since the
caller can still just cancel, but the same is true for circular payments).

Yes, I am not so sure myself. A second operation with some sort of cryptographic relationship might be necessary. I also wonder if it is possible to design for a sender, Alice, to inform recipient, Bob, that there is some space in the onion that Bob is free to stomp on with new data addressed to Carol of Bob's choosing. That would scramble any checksumming and perhaps leak metadata if someone observes the 'before' and 'after' packet, so it might not be kosher for Sphinx (I can't clam to understand the exact Sphinx packet details in depth).

Definitely agree with this one, I think we can build some very idiomatic
abstractions on top of the raw RPC, and we're just scratching the surface
here. I have been playing with circular payments a bit lately and we might
subsume some of the common patterns into a pythonic library built on top of
the raw RPC.

Sounds good. Figuring out the recipe for a circular route - which I ended up taking logic from sendinvoiceless.py rather than trying to write my own - was a stumbling block for getting started. Also, there was a gotcha with setting the CLTV values properly which caused some operations to fail intermittently. A well-written primitive in the common infrastructure would have definitely saved me several hours.

I think this use-case would have been better served using the keysend
protocol, allowing the plugin to immediately decide whether to draw or not
based on whether it can terminate the payment or it has to fail it. This would
have removed the caching of actions and the chances of failing even after
having cached the actions.

I am inclined to agree. It's not a huge deal for the Onion Studio tech demo, however a real economic transaction might benefit from some sort of proof-of-payment from getting an invoice including a payment_hash signed by the service, which is lacking with KeySend. The circular payment lacks this too since the invoice is self-generated. Here's an area where a more complicated mult-part TLV operation can perhaps satisfy all the requirements.

I think we have a couple of people asking for more fine-grained HTLC
notifications, so that might indeed be implemented some time soon. See
tracking issue #3424. The major question is what kind of payload should these
have? I don't think that passing in the onion upon each call would be
sensible: it's 2.6 kB of hex encoded binary data that we'd pass in 2 times,
and for forwards we'd add it 4 times in total (2 on the incoming side, and 2
on the outgoing side). I'd say we can have the onion in the htlc_added case,
and the plugin can cache the onion, being certain to get notified once it is
no longer of interest and can be cleared. Does that sound like a good
middleground?

Cool. Thanks for filing the issue.

My first thought is that the non-routing TLVs found in the payload could be decoded and passed in a JSON form to the plugin - perhaps mirroring the JSON form being passed in to an improved onion/payload creator. The advantage might be to avoid needing the BOLT 1 encoding/decoding logic plugin-side and duplicated across all plugin languages.

On the other hand, on passing the whole 2.6 kB of hex - Might it be desirable to allow a clever scheme to hide data in the 'spare' sections of an onion that is only revealed by the knowledge of a preimage or something else entirely? Might be an argument to pass the whole thing to allow such experiments.

This should already be possible with the current protocol, just encrypt a
field in the payload using the preimage on the sender side, and then wait for
the HTLC to be settled.

Gotcha. Thanks. I had the brief thought of wanting to protocol-ize it perhaps provide the decryption automatically via the HTLC lifecycle notifications to avoid the plugin needing to do its own encrypt/decrypt operations. However, this is probably overthinking it, lacking a concrete use case, so I don't feel strongly about that right now.

I have been talking with some other plugin devs about creating a Payment class
in the python library that can be inherited/customize to add custom behavior,
but implements the generic functionality to find alternative routes and retry
when a recoverable error is returned. This class could then be customized to
pull pixels from a queue of pixels to draw, packing as many of them into a
single payment as will fit, retrying if failed.

Sounds like a good abstraction to me. The basic 'pull pixels from queue' loop I have implemented in my code here

I think if there were this Payment class, I might want to inherit my Draw class from it. Since Draw needs several/many payments to all go through, it might desire some estimation in the Payment class to do a rough estimation of whether it can all go through given the current channel capacity state of the network. I assume Payment would already have to be AMP/MPP aware, so a similar heuristic might also be able to estimate whether N small sequenced payments can also go through.

I think AMP is likely not what you want here, since you want pixels and
payments to coincide. Your problem is pretty orthogonal to what AMP tries to
achieve, and I'd think that your amounts are small enough not to require
splitting into parts. A more standardized fail-retry-loop is more likely what
you need here :-)

Could be a reach on my part, but what I am ultimately desiring is a way to deliver the mentioned 160 data payloads and associated payments atomically. The obvious downside is that it would consume a lot of network resources to have lots of little pending operations open at the same time. The one-at-a-time retry loop is the way around that problem at the expense of taking a long time to send all the pixels.

Down the road, perhaps a scheme could be designed where there is some smaller payments to 'rent' the middle-node resources for attempting a larger operations which might or might not close atomically. Probably requires a bunch of thought and design, but I figure is is worth bringing up since the problem is observable in Onion Studio.

~

happy to close this issue if it is quiet for a few days.

[ZmnSCPxj]

I also wonder if it is possible to design for a sender, Alice, to inform recipient, Bob, that there is some space in the onion that Bob is free to stomp on with new data addressed to Carol of Bob's choosing.

I believe not, but only because of the need to preserve the same ephemeral key throughout the onion. If we could switch ephemeral keys, this might work, but I am uncertain about the security of that; I think @cdecker considered this before and concluded it was insecure for reasons beyond my ken.

Figuring out the recipe for a circular route - which I ended up taking logic from sendinvoiceless.py rather than trying to write my own - was a stumbling block for getting started.

It seems to me that factoring out a makecircularroute into a separate plugin that this one and sendinvoiceless.py is dependent on would be good.

It's not a huge deal for the Onion Studio tech demo, however a real economic transaction might benefit from some sort of proof-of-payment from getting an invoice including a payment_hash signed by the service, which is lacking with KeySend. The circular payment lacks this too since the invoice is self-generated. Here's an area where a more complicated mult-part TLV operation can perhaps satisfy all the requirements.

With payment points+scalar this becomes possible to embed in a non-final onion hop, because homomorphisms.

[cdecker]

Thanks! Just from discussing and spitballing some ideas in various channels
over the last few days, it is great to see that the ideas are starting to flow
out there. I am feeling that the new-style tightly-coupled "payment +
instruction" mode of interaction with a remote node kind of throws a wrench
into some of the initial wave of thought for how LN UIs might turn out:

type words, add to cart, request invoice, pay invoice becomes ---> key press sends payment along with key

Agreed, my hope is that it can eventually be like this:

1. Make your selection of actions to perform
2. In parallel the invoice updates in real-time
3. Eventually you click the pay button that is nicely integrated into the web page
4. A URI handler waiting for lightning:lnbt123... URIs gets triggered
5. Depending on the settings of the wallet the amount may be low enough to be automatically paid, otherwise a confirmation is asked
6. Fin

I would love to continue work on collaborating on infrastructure and the
larger LN and C-Lightning project in particular, but unfortunately, I can't
commit to being responsive and reliable and don't want to slow anyone
down. The progress and speed of this project is amazing and I simply don't
have the spare cycles to keep pace in the way that I would want to. I am
subscribed and reading most of what goes by here, and am glad to chip in as I
can, though.

Don't worry, every bit counts. No matter whether it's the sporadic
contribution from a hobby project or a fulltime job. Your effort of building
onion.studio, and writing up your experiences is very helpful for us :-)

In a scheme like that, Alice would have to estimate 3 different routes with
different data requirements before deciding to send. Thinking along those
lines, my suggestions for something 'nice' to provide in a route
creator/estimator might still be premature. It might be better to understand
more concrete examples of cool ideas first.

I can probably provide you with a formula to estimate the available space, let
me give it a shot. The constraints are as follows:

sum(payload_lengths) + num_hops * 32 bytes hmacs <= 1300

Payload in this case refers to whatever you add to the call to
createonion. The length of the payload is a bit more involved (33 bytes for
legacy, variable for TLV payloads). We could provide a length computation in
the onion library if that makes things easier.

I'll be perfectly honest, I didn't connect that LND's KeySend was possible in
the current C-Lightning release via response to the htlc_accepted hook until
I read it in the noise code the other day. My mind was still pegged to a
narrower understanding. I guess this is how I learn. :)

No problem, we haven't publicized it widely just yet :-)

Examining the new --data and --keysend options in the LND implementation,
it looks like the command set only currently allows you to put a payload on
the final hop. That strikes me as, on one hand, very pragmatic for getting
started with the first wave of message-sending apps, but, on the other hand,
is not quite satisfying the same programming power that has captured my
interest.

I guess imitation is the sincerest form of flattery :-)

I guess I am still in the anti-pragmatic mode of playing about trying to
understand the larger set of potentials. At least for now, circular payments
have a bit more to talk about. Also, I can report that as a result of
whiteboarding the explanation circular payments for Onion Studio, I have also
caused a few "ahah" moments for people understanding channel rebalances and
how the sendinvoiceless.py plugin works.

Absolutely, I had not considered the "hide a message in a longer route"
aspect 😉

Yes, I am not so sure myself. A second operation with some sort of
cryptographic relationship might be necessary. I also wonder if it is possible
to design for a sender, Alice, to inform recipient, Bob, that there is some
space in the onion that Bob is free to stomp on with new data addressed to
Carol of Bob's choosing. That would scramble any checksumming and perhaps leak
metadata if someone observes the 'before' and 'after' packet, so it might not
be kosher for Sphinx (I can't clam to understand the exact Sphinx packet
details in depth).

Not with the onion construction we have today, since the entirety of the onion
is enforced by the HMAC, any change in the onion would break integrity and the
next node would reject it. I don't see a way to work around telling the server
your identity.

My first thought is that the non-routing TLVs found in the payload could be
decoded and passed in a JSON form to the plugin - perhaps mirroring the JSON
form being passed in to an improved onion/payload creator. The advantage might
be to avoid needing the BOLT 1 encoding/decoding logic plugin-side and
duplicated across all plugin languages.

The issue is that it'd be an incomplete solution at best, since lightningd
doesn't know what type the value of the TLV is, and so we'd always be passing
the hex-encoded value, no matter whether it's supposed to be an integer, a
hash, a pubkey, or a variable integer. So we can do one small step towards
decoding TLVs, but we can't do all the work 😉

On the other hand, on passing the whole 2.6 kB of hex - Might it be desirable
to allow a clever scheme to hide data in the 'spare' sections of an onion that
is only revealed by the knowledge of a preimage or something else entirely?
Might be an argument to pass the whole thing to allow such experiments.

Not really possible, since the entirety of the onion is deterministically
generated, onion encrypted and HMACd, No way to fiddle around with the unused
part of the onion :-)

I think if there were this Payment class, I might want to inherit my Draw
class from it. Since Draw needs several/many payments to all go through, it
might desire some estimation in the Payment class to do a rough estimation
of whether it can all go through given the current channel capacity state of
the network. I assume Payment would already have to be AMP/MPP aware, so a
similar heuristic might also be able to estimate whether N small sequenced
payments can also go through.

I was thinking about implementing the getroute + sendpay + retry loop
initially and then we can add more smarts via mixins and inheritance. The
first improvement on this would indeed be MPP in a subclass.

Down the road, perhaps a scheme could be designed where there is some smaller
payments to 'rent' the middle-node resources for attempting a larger
operations which might or might not close atomically. Probably requires a
bunch of thought and design, but I figure is is worth bringing up since the
problem is observable in Onion Studio.

Why not deliver pixels with their associated pixels, so if you have 1000px,
costing 1'000'000 msat, you can split that into, say, 25 parts each of 40'000
msat and 40px in the payload (potentially each with its own payment_hash)
and a reference to the overall action). The plugin can then accumulate the
parts, accepting the payment and drawing once all the parts are received.

This is basically emulating MPP, but you could do partial draws if the full
thing is taking too long (and free the HTLCs along the routes). You'd not
associate all the parts with each other (different payment_hashes, which in
MPP we are still trying to get working).

I believe not, but only because of the need to preserve the same ephemeral key
throughout the onion. If we could switch ephemeral keys, this might work, but
I am uncertain about the security of that; I think @cdecker considered this
before and concluded it was insecure for reasons beyond my ken.

I still am considering this, but it doesn't allow modification of the existing
payload at intermediate steps. It works for rendez-vous routing (which is the
context I am working on this), but doesn't work for intermediate nodes adding
information, due to the HMACs mentioned above: if the original creator of the
onion doesn't know the full onion at each hop it can't generate the HMACs
ensuring integrity.

It seems to me that factoring out a makecircularroute into a separate plugin
that this one and sendinvoiceless.py is dependent on would be good.

@renepickhardt has recently opened a pull-request on the plugins
repo implementing what he calls a fixroute (but I think of it more
of a waypoint route) in which you can specify waypoints that a route should
go through. Pretty sure that'd cover this use-case pretty nicely 😉