BOLT 1: Add a TLV stream extension to existing messages.

[t-bast]

The spec already prepared a hook to add additional information to existing
messages (additional bytes at the end of a message must be ignored).

Since we're using TLV in many places, it would make sense to use that optional
additional space at the end of each message to allow an optional tlv stream.

We keep the "additional bytes must be ignored" rule for what happens after that
tlv stream, giving us extra flexibility to add another type of optional content
to existing messages later if needed.

[rustyrussell]

I think not yet. While we're likely that all future extensions will be TLV streams, it's still best to be explicit where they are used IMHO.

[niftynei]

Another consideration (which builds on @rustyrussell's point about being explicit where TLV's are used) is that TLV type 'namespacing' currently is defined as being unique to the message that they're defined for.

The `type` is a varint encoded using the bitcoin CompactSize format. It
functions as a message-specific, 64-bit identifier for the `tlv_record`
determining how the contents of `value` should be decoded.

How would extension types be reserved?

[t-bast]

Yeah I agree that for each message, we need to define the associated tlv types.
So it's probably better to be explicit anyway as you suggest.
I wanted to take this opportunity to add this rule to allow us to start experimenting on mainnet by appending tlv streams to existing messages, but I guess we can do that anyway since the trailing bytes currently must be ignored for all existing messages.
Thanks for your feedback!

[dr-orlovsky]

While we are understanding a point made by @rustyrussell, here is the case we'd like to ask you to consider. At @rgb-org we are developing RGB, a technology to issue assets at L2 using Bitcoin with minimal blockchain pollution and client-side validation. It includes Spectrum, a L3 technology for assets that will be able to work with LN and will be 100% BOLT-compatible. You can check https://github.com/rgb-org/spec/blob/spectrum-0.5/04-lightning-network.md for details.

In order to achieve our goal we need to add TLV extensions to init, open_channel, accept_channel, update_add_htlc and probably some other messages to add an information on the assets included into the channel. While we can wait until TLV extensions will be added into each of those messages, it will (a) take a long time and (b) we are not sure that all messages required for Spectrum will get this extensions. So I'd like to ask you to reconsider your decision: adding generic TLV extensions to all message types will help not only our project, but many other L3 projects out there.

[Roasbeef]

@dr-orlovsky before discussing any possible extensions to the specification, there should first be a comprehensive specification on exactly how RBG wishes to integrate colored coins into regular channels.

I took a glance at the document you linked above, and it's still unclear to me: exactly how assets are tracked along side regular HTLCs, how they're tracked within the settle balance of each channel, if it's possible to have multiple assets in a settled balance, how one ensures that the same asset will be exchanged along the entirety of the route, how fees are paid for pure asset transactions (and also in which currency). Those are just a few questions that popped up at a glance of the document.

[dr-orlovsky]

Thank you, @Roasbeef, for the elaboration on the questions that has to be covered. Let me provide quick answers (most of which is coming from the main part of the RGB specification https://github.com/rgb-org/spec/blob/rgb-0.5/01-rgb.md, since LN part is just re-using it)

1. How assets are tracked along side regular HTLCs and how they're tracked within the settle balance of each channel

They are tracked using single-use seals mechanism in the same way they are tracked for non-LN part; namely, nodes will agree on the new channel balance, which includes both bitcoin and asset balances (for all assets involved), which can be transferred via TLV extensions to update_add_htlc message and outside on LN-based communications. Asset balances are kept in the proofs, which are cryptographically committed using pay-to-contract commitment to P2PWKH output of the commitment tx. You can see more details in the following parts of the spec:

• https://github.com/rgb-org/spec/blob/rgb-0.5/01-rgb.md#overview
• https://github.com/rgb-org/spec/blob/spectrum-0.5/04-lightning-network.md#changes-to-specific-messages

2. if it's possible to have multiple assets in a settled balance

For sure, it is possible. Assets are settled off-chain, with a specially-designed proofs, as described above

3. how fees are paid for pure asset transactions (and also in which currency)

There are no "pure" asset transactions, since each transaction involves movement of some above-dust_limit satoshis; which are used to pay the fees in the normal way. Actually non-Spectrum enabled nodes just don't know that there is some asset movement happening.

While I understand that these points can be hard to grasp from the first read, and the spec regarding LN part is still a WIP, it's important to understand the possibility and a timeline for TLV inclusion into LN messages, since we need to understand is this way opened for us – or we need to undertake some other. The technology is having been discussed and developed for two years by the team of ~15 ppl, including Giacomo Zucco, Peter Todd and others, and there are plans by Bitfinex to launch USDT on it (https://www.coindesk.com/startups-debut-first-protocol-for-issuing-tokens-via-bitcoins-lightning-network). So extending messages with ability to pass arbitrary data is really important for the success of this and other L3 projects, and I will highly appreciate any clarity regarding this question...

[Roasbeef]

Asset balances are kept in the proofs, which are cryptographically committed using pay-to-contract commitment to P2PWKH output of the commitment tx

Are you aware of #642? After this, the default channel type's non-delay output will no longer have any sort of randomness mixed with it, and will be static throughout the lifetime of the channel.

There are no "pure" asset transactions, since each transaction involves movement of some above-dust_limit satoshis; which are used to pay the fees in the normal way.

If this is the case, how do I signal I want to trade my beefcoins for osuntokens when I traverse the link A <-> B. How can this work if the incoming link doesn't have any beefcoins, are they "magically" created at that point?

When I open a channel, and want to have a balance of 100% beefcoins, how does the responder know that those are valid beefcoins created by Palace of Prime Rib?

[dr-orlovsky]

Are you aware of #642? After this, the default channel type's non-delay output will no longer have any sort of randomness mixed with it, and will be static throughout the lifetime of the channel.

No, I did not, thank you for the information! Fortunately while remote_key was deterministicly derived we were able to tweak it, but for sure its better to have a static version of it. BTW, in order to create asset-enabled channels, both parties have to signal support for the same assets, this is why non-static remotekey was not a big problem. This brings us to your next question:

If this is the case, how do I signal I want to trade my beefcoins for osuntokens when I traverse the link A <-> B. How can this work if the incoming link doesn't have any beefcoins, are they "magically" created at that point?
When I open a channel, and want to have a balance of 100% beefcoins, how does the responder know that those are valid beefcoins created by Palace of Prime Rib?

Nothing can be issued ("magically created") inside the LN. You can (a) pass some assets or (b) trade them with BTCs, so if you want to trade for instance AAPL shares for USDTs, you need to first trade AAPL for BTC, and then BTC for USDTs. The incoming link must have AAPL to trade and it must provide you with a proper proof of the ownership, as described in the main RGB spec https://github.com/rgb-org/spec/blob/rgb-0.5/01-rgb.md#overview

For instance, if Alice wants to sell Bob some assets for BTC,

1. Alice should establish Spectrum-enabled channel with Bob in the following way:
   • when connected, both add to the init message feature bit 10 signalling optional Spectrum support, and provide a list of available assets in TLV extension to the init message. Alice specifies that she can allocate X of asset A into the channel, and Bob indicates that he is willing to exchange in assetA/BTC pair with some price.
   • next, Alice sends open_channel message with another TLV extension indicating that she will be adding X of asset A to the channel (since now she knows that Bob is willing to be liquidity provider) and adds a proof history to TLV in a serialized form (or, for a large proof history, some URL or P2P/Tor network address to download it from).
   • Bob responds with accept_channel message confirming in its TLV extension that he is agreed to have X of asset A shares in the channel from the Alice side
2. Next, once the channel is established, instead of sending Bob update_add_htlc message with spending BTCs, it will be Bob sending Alice BTC in exchange for assets, so
   • Alice sends Bob update_add_htlc message with a TLV extension specifying that the channel update should include transfer of X of asset A;
   • both create a new proof for asset A asset transfer from Alice to Bob (the proof is deterministicly constructed, since Bob now has the full proof history from Alice – see above pt. 1)
   • the new commitment transaction is also constructed by both parties, that includes a (a) new balance (with some BTC from Bob going to Alice) and (b) tweaked public key in the second output (which is P2WPKH paying to remote pubkey), committing to the newly constructed proof with pay to contract commitment
   • the rest of the update happens as usual

PS. Our hopes that the protocol will be primarily used for more classic digital assets (like corporate shares, bonds, collectibles and stable currencies alike USDT) rather then creation of new coins. I know that we can't force people not to scam, but "fortunately" they have more simple ERC20 on Ethereum, so while its (yet) alive, we a kind of "protected" :) :) :)

PPS. Are you sure that this is a right place to explain all the details of RGB/Spectrum functionality? I'm just afraid to flood the PR with off-topic information. The point which is relevant to this closed PR is the following: there is a business case requiring TLV extensions to some set of messages, and since it can take a lot of time to add TLV to all messages that we need to, I'm asking is it possible to re-open this PR and merge it into the spec?

[dr-orlovsky]

@Roasbeef here is another sample of L3 tech that will clearly benefit (and will require) TLV extensions to LN messages: https://github.com/storm-org/storm-spec

This is distributed storage and messaging with economic incentivisation leveraging LNP/BP ecosystem, like FileCoin, Ethereum Swarm etc, but w/o tokens/shitcoins, addressing the same problem with just pure Bitcoin and LN. It works using special type of payment channels guaranteeing remote data storage and retrieval with counterparty risks mitigated by economic stimulus (stakes etc). More importantly, these channels can be embedded into LN channels, but in order to achieve this, we first need to modify the structure of the commitment transaction, and then insert additional data on its update into update_add_htlc messages.

I'm writing this just to emphasize that addition of generic TLV extensions will unlock a LOT of potential applications for LN outside of simple direct or routed payments :)

[t-bast]

@dr-orlovsky thanks for this discussion, it's good to know what you are building in your layer 3 proposal and feed that back into something coherent in layer 2.

We are completely open to defining that the messages you mention (update_add_htlc and a few others) should optionally contain a trailing tlv stream. The reason this PR was closed was because it was too general. What we need to follow up would be a proposal for the tlv types that would be added to each specific message. Tlv type numbers need to be reserved and included in the spec (we can't have a collision with two different apps using the same tlv number for different things otherwise the network would split).

If you could submit a PR to define the tlv types you need for the messages you want to extend, that would get the ball rolling and would make it easy to evaluate when/if that change is acceptable.

[Roasbeef]

(FWIW, this branch has been deleted, so it actually can't be re-opened @dr-orlovsky)

What we need to follow up would be a proposal for the tlv types that would be added to each specific message. Tlv type numbers need to be reserved and included in the spec (we can't have a collision with two different apps using the same tlv number for different things otherwise the network would split)

@t-bast I'm not sure I agree that this PR was too general. Light coordination w.r.t global feature bits should be enough to avoid any collisions in the future. What we need is a way for developers to experiment with slightly different channel type or sub-protocols, without needing to walk through the entire spec process. Also in many cases, it may be far too early if the protocols are still evolving. For example, referencing the "storm" application above, are we really going to have a full specification for PCPs in the near term?

It would be better (less coordination all around) for developers to know what ranges they can utilize, then use a lightweight announcement process on the mailing list to declare which ranges they're using for experiments.

[t-bast]

This is interesting and seems to be a recurring point, as #623 and #660 show.
It could be a good idea to make progress on that subject and find the sweet spot that allows easy experimentation without impacting mainnet negatively.

[dr-orlovsky]

referencing the "storm" application above, are we really going to have a full specification for PCPs in the near term?

It will clearly take some time, since it will be discussed with the community and there certainly will be improvements and changes. It's pretty much like with Lightning itself, so I do absolutely agree with you that flexibility for devs is required.

It could be a good idea to make progress on that subject and find the sweet spot that allows easy experimentation without impacting mainnet negatively.

May it it could be possible to define a set of primitive types, and the rest to be composible, not requiring to be pre-registered? For instance a node may provide a special message in the init protocol to define all those composed types it uses.

However, what is even more important, is how to handle the situation if some lightning node has multiple plugins/extensions, with each adding some data to the same message.