Bolt 1: Specify that extensions to existing messages must use TLV #714

t-bast · 2019-12-09T10:28:40Z

The spec already prepared a hook to add additional information to existing
messages (additional bytes at the end of a message must be ignored).

Since we're using TLV in many places, it would make sense to use that optional
additional space at the end of each message to allow an optional tlv stream.

We keep the "additional bytes must be ignored" rule for what happens after that
tlv stream, giving us extra flexibility to add another type of optional content
to existing messages later if needed.

This was already proposed in #630 but rejected. I found additional reasons to
specify this now rather than on a per-message basis. We don't want to have two
competing and incompatible ways of extending existing messages.

Imagine node A and node B are peers, both running experiments by leveraging the
"ignore additional data" rule. However node A uses a TLV stream (as proposed in
this PR) while node B uses a different encoding. Unluckily node B also started his
encoding with something that can be interpreted as a varint, so node A interprets
this as a TLV stream, but it decodes as a two-bytes even number (which has a 50%
chance of happening). She (validly) thinks it's an even TLV she doesn't understand,
so she fails the connection and closes the channels.

This is a major inconvenience when experimenting with message extensions. I believe
what this PR proposes is reasonable and prevents these competing encodings to mess
up experimentations.

t-bast · 2019-12-09T12:06:07Z

Open question: should the TLV stream be prefixed with its length or not?

It seems like the gossip_queries tlv streams today don't prefix with the length (length is inferred from the overall message length).

The inconvenient when we don't prefix with the length is that we completely remove the possibility of having another bag of uninterpreted bytes at the end of the message (if we decide some day that we want to extend messages with something else than TLV). Maybe we don't care, maybe we do :)

I'm leaning towards removing the tlv stream length prefix: in that case this PR makes even more sense to do now, because we don't want to have some messages be extended with a TLV stream and others extended differently, that would be a big mess. WDYT @rustyrussell @cfromknecht @sstone ?

UPDATED: removed the length prefix in 6117c30.

cdecker · 2019-12-09T18:54:50Z

If we limit message extensions to be TLV-based we should definitely drop the length prefix.

I'd like to point out that there is a small caveat for gossip broadcast messages (node_announcement, channel_announcement and channel_update) which requires that they include the fields in the signature, which is the main reason we went for a canonical encoding for TLVs (no duplicate types and monotonically increasing type order). Just as a caveat to the ignore word that is being used in the PR.

Other than that small wording issue, I'm definitely in favor of consolidating towards TLVs.

t-bast · 2019-12-10T10:12:32Z

I added two commits:

6117c30 simply removes the length-prefix as agreed during the spec meeting
274a965 makes mandatory some message optional fields that would otherwise make parsing the extension non-deterministic: the commit message contains additional details. @Roasbeef @cfromknecht let me know if making these fields mandatory seems ok to you (thanks for raising the issue during the meeting).

t-bast · 2020-01-23T13:47:16Z

Do you think we should re-work upfront_shutdown_script to move it to a TLV record instead of the current optional encoding? That would simplify the migration to TLV for all messages.

It could be done in an interesting backwards-compatible way:

assign type 0x00 to upfront_shutdown_script
then for all scripts that have a length < 253 the TLV encoding will perfectly match the current encoding

halseth · 2020-01-30T07:30:51Z

Do you think we should re-work upfront_shutdown_script to move it to a TLV record instead of the current optional encoding? That would simplify the migration to TLV for all messages.

It could be done in an interesting backwards-compatible way:

assign type 0x00 to upfront_shutdown_script

then for all scripts that have a length < 253 the TLV encoding will perfectly match the current encoding

Sounds easiest just to make it a mandatoy field? (similar to what this PR specifies now) Why would doing this "hack" make the migration to TLV simpler?

Is upfront_shutdown the only optional field across all wire messages that interfers with the TLV stream?

t-bast · 2020-01-30T08:54:04Z

Thanks for the comment @halseth.

Sounds easiest just to make it a mandatoy field? (similar to what this PR specifies now) Why would doing this "hack" make the migration to TLV simpler?

If we change the format of open_channel to be (and do a similar thing for accept_channel):

type: 32 (open_channel)
data:

[chain_hash:chain_hash]
...
[byte:channel_flags]
[open_tlvs:tlvs]

tlvs: open_tlvs
types:
1. type: 0 (upfront_shutdown_script)
2. data:
  - [...*byte:script]

Then we have a smooth upgrade path and we clean-up the spec by moving this field inside a TLV stream, because it will result in the exact same encoding as today. Since valid scripts have a length < 253, they will be encoded as either 0x0000 (when empty) or 0x00021234 (for a length 2 script 0x1234). Because of TLV streams canonical encoding, this will be the first record to be encoded. This allows us to either include or omit the upfront_shutdown_script field (whereas simply making it mandatory means we'd always be including it, even if it's an empty one 0x0000 which is a bit ugly). And it allows upgraded node to correctly decode data from non-upgraded nodes (in both cases where the field is present or absent) and encode data that non-upgraded nodes will understand (because they should ignore unknown trailing bytes).

Does that make sense? The more I think about it, the less hackish it seems. I think it nicely cleans up that field that otherwise feels a bit out of place, being optional but not a TLV.

Is upfront_shutdown the only optional field across all wire messages that interfers with the TLV stream?

It's the only one that's a bit painful to handle. See 274a965 for details (especially the commit message). The two other optional fields are quite easy to simply make mandatory or not touch.

The spec already prepared a hook to add additional information to existing messages (additional bytes at the end of a message must be ignored). Since we're using TLV in many places, it would make sense to use that optional additional space at the end of each message to allow an optional tlv stream. We keep the "additional bytes must be ignored" rule for what happens after that tlv stream, giving us extra flexibility to add another type of optional content to existing messages later if needed.

Clarify that extension bytes are included in signatures.

This applies to: - option_upfront_shutdown_script: if you're not interested, just set the length to 0 - channel_reestablish commitment points: it makes sense to always include those regardless of whether `option_dataloss_protect` or `option_static_remotekey` are set No need to change the `channel_update`'s `htlc_maximum_msat` because the `message_flags` encode its presence/absence. It can still be either included or omitted without causing issues to the extension stream.

@TheBlueMatt

@TheBlueMatt made a point that always parsing a TLV stream even when you don't expect anything is cumbersome. If you don't expect any extension, you may ignore the tlv stream. @rustyrussell noticed an unnecessary requirement now that TLV stream isn't length-prefixed.

This makes more sense for future work. It gives us more room to change the encoding or remove it.

t-bast · 2020-02-04T13:44:59Z

I made upfront_shutdown_script a TLV record in commit db31eaf.

If you all prefer, I can instead simply make the field mandatory, but please do consider the TLV option.
Having it as a TLV record allows us to later remove the sender requirement of always encoding an upfront_shutdown_script, or to change the preferred way of encoding it.
Having it as a mandatory field outside of the TLV stream means we can never get rid of it in the future.

Here are all the cases listed to show that backwards-compatibility fully works (but honestly you can easily convince yourself without going through all examples).

With `option_upfront_shutdown_script` set by both nodes

Legacy node including a `shutdown_script`

For example a P2WSH: 002011418a2d282a40461966e4f578e1fdf633ad15c1b7fb3e771d14361127233be1.
The encoded trailing bytes (after channel_flags) are 0022002011418a2d282a40461966e4f578e1fdf633ad15c1b7fb3e771d14361127233be1.

Updated nodes interpret that as:

00: TLV type (upfront_shutdown_script)
22: length
002011418a2d282a40461966e4f578e1fdf633ad15c1b7fb3e771d14361127233be1: shutdown_scriptpubkey

Which works with the TLV definition.

Legacy node including an empty `shutdown_script`

The encoded trailing bytes (after channel_flags) are 0000.

Updated nodes interpret that as:

00: TLV type (upfront_shutdown_script)
00: length
``: shutdown_scriptpubkey

Which works with the TLV definition.

Updated node including an empty `upfront_shutdown_script` TLV

The encoded trailing bytes (after channel_flags) are 0000.

Legacy nodes interpret that as:

0000: shutdown_len

Which works with the legacy definition.

Updated node including an `upfront_shutdown_script` TLV

For example a P2WSH: 002011418a2d282a40461966e4f578e1fdf633ad15c1b7fb3e771d14361127233be1.
The encoded trailing bytes (after channel_flags) are 0022002011418a2d282a40461966e4f578e1fdf633ad15c1b7fb3e771d14361127233be1.

Legacy nodes interpret that as:

0022: shutdown_len
002011418a2d282a40461966e4f578e1fdf633ad15c1b7fb3e771d14361127233be1: shutdown_scriptpubkey

Which works with the legacy definition.

Updated node including additional TLVs

For example with an empty upfront_shutdown_script, but would work the same with a non-empty one.
The encoded trailing bytes (after channel_flags) are 0000 01012a (TLV record 1).

Legacy nodes interpret that as:

0000: shutdown_len
01012a: ignored trailing bytes (Bolt1: MUST ignore any additional data within a message beyond the length that it expects for that type)

Which works with the legacy definition.