-
Notifications
You must be signed in to change notification settings - Fork 512
Flat features: far simpler solution for feature bits. #666
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Flat features: far simpler solution for feature bits. #666
Conversation
01-messaging.md
Outdated
|
||
#### Requirements | ||
|
||
The sending node: | ||
- MUST send `init` as the first Lightning message for any connection. | ||
- MUST set `mbz` to zero. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: should we mention that it's for backwards-compatibility?
Crazy idea (but could be done later): why not:
- [
u16
:mbz
] - [
u16
:mbz
] - [
tlvs
]
And define a first tlv type (2
?) to hold feature bits?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not backward compatible. unfortunately. Currently we have no deployed global bits, so we can do the mbz trick.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We do have a global feature bit spec-ed (variable-length onion).
For eclair we haven't made a release that advertises it yet, but if people deployed nodes from master then they will be sending something in the global features field.
So our code would need to handle both the previous format and the new with mbz
(at least for a while)...this isn't too painful to implement though (just branch on mbz
being 0
or not).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, since it's not deployed I was hoping we could sidestep this... but perhaps not.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What does mbz
mean here? "Must be zero"?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We haven't cut a release either using the new global feature bits, but are very close to doing so....
As a result, would be nice to have all this clarified as otherwise with our target release schedule, we may cut a version that still uses the existing global feature bits.
Stepping back for a second, this mbz
change doesn't really strike me as being backwards compatible. We're potentially fracturing the network here (let's say people are using a fork of some implementation in the wild and are using global feature bits to gate their protocol). Instead we can simply rename the field in the spec: global isn't bound to anything (don't set a name), and local becomes the features
as is here. For some period of time, we should still also merge the two fields together.
09-features.md
Outdated
| 3 | `initial_routing_sync` | Sending node needs a complete routing information dump | I | [BOLT #7][bolt07-sync] | | ||
| 4/5 | `option_upfront_shutdown_script` | Commits to a shutdown scriptpubkey when opening channel | IN | [BOLT #2][bolt02-open] | | ||
| 6/7 | `gossip_queries` | More sophisticated gossip control | IN | [BOLT #7][bolt07-query] | | ||
| 8/9 | `var_onion_optin` | Requires/supports variable-length routing onion payloads | IN | [Routing Onion Specification][bolt04] | |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Shouldn't this be INC
?
This could allow nodes to force some of their channels to use variable-length onions (and explicitly disallow the legacy fixed-frames) by setting bit 8 in the channel_update
of such channels.
Not sure whether it's really useful or not, but maybe worth discussing?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It's tempting, since node_announcement propagation is currently less reliable than channel_announcements. But channel_announcements can't be updated, so current channels would never set this bit, even though they'd be perfectly fine.
So I think it needs to be a Node bit.
Why aren't the three scenarios in this PR body expressible using the current set of feature bit fields and names? We can just leave the existing local feature bits as is, but then start to define all new bits as a global feature bit. The node and channel announcement fields would then only inherit the name space from the global feature bits. Even in this latest iteration, I still have a nagging feeling that this isn't really fixing anything. |
(also this PR is evil, just saying) |
Because everyone got confused about the two schemes, yet a single scheme also sucks, because optional and compulsory are different in different contexts:
TLDR: much easier to think about what each feature needs, and specify it at that level. |
926e960
to
2f5176e
Compare
Rebased, restored gflen field. |
It would be nice to make quick progress there; this way we're not slowing down releases. |
The Context column decodes as follows: | ||
* `I`: presented in the `init` message. | ||
* `N`: presented in the `node_announcement` messages | ||
* `C`: presented in the `channel_announcement` message. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What about the channel update message? We also need possibly extend the existing hop hints to also encompass this information. One could say the current set of feature bits handles that, however it's possible to provide multiple hop hints, and each of those channels could themselves advertise/require distinct sets of hop hints.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We don't have a general feature field in channel_update (yet?). But we can add a field if we need to.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think the main thing here is the inability to express distinct feature sets for each channel with the current BOLT 11 feature bit vector extension.
Our release is late as is, and it's now being held up by this PR as it creates a dependency with the static remote key PR. We're now schedule to tag next week. As for the current state of the global feature bits: the static key PR is more or less finalized/ready but is now blocked on the inclusion of this PR, the onion tlv feature has itself long been included. |
| 3 | `initial_routing_sync` | Sending node needs a complete routing information dump | I | [BOLT #7][bolt07-sync] | | ||
| 4/5 | `option_upfront_shutdown_script` | Commits to a shutdown scriptpubkey when opening channel | IN | [BOLT #2][bolt02-open] | | ||
| 6/7 | `gossip_queries` | More sophisticated gossip control | IN | [BOLT #7][bolt07-query] | | ||
| 8/9 | `var_onion_optin` | Requires/supports variable-length routing onion payloads | IN | [Routing Onion Specification][bolt04] | |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why should a node level feature gate if two peers can/will-continue-to communicate on the connection level? As is, if I went to open a channel with a peer that doesn't support the new onion payload, but they required this bit, then the connection would fail upon feature bit comparison.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, it's overbroad: this is the tradeoff. Previously you could in theory gossip with a node which had an even global feature you didn't like, too.
Putting it in the channel_announce would also be overbroad (since both nodes might not need it). We could add a feature bitfield (TLV-style) to channel_update, but it's marginally more efficient to use node_announce, since it's really per-node not per channel in any sane implementation.
Did we want to go there?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
var_onion_optin
does not affect the p2p layer in lnd at all, does it in other implementations? is the rationale is it mainly to decide whether the node supports TLV onions during channel creation?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We're not using it either for p2p, it will currently only be useful in the Bolt 11 invoice or inferred by other feature bits (MPP, Trampoline, etc).
At this point, we may need to just accept the existing global feature bit namespace, then once this is ready and tested for interop all around, copy over the existing global feature bit namespace to what is currently known as the local feature bit namespace. However, re the combination of the namespaces, see my latest comment re cross-layer feature bit awareness comparisons. |
To remedy that comparison issue, we may need to add additional feature bit vectors maybe? Not sure this is really doable though, since the |
Yeah, I might have been too optimistic by hoping we could merge this soon to unblock your release from further delay :). I quite like the current proposal. Updated writers will only write to the currently-named Legacy readers will not be impacted because right now the only global feature bit (variable length onion) will only be set as optional, not mandatory (they aren't missing out on anything since we don't have real features like AMP that actually leverage the variable length onion). We can have our implementations handle both the legacy case (if we see that |
There's definitely no harm in advertizing var_onion_optin in local features, and I'd definitely recommend it. The other questions are:
|
This preempts the acceptance of lightning/bolts#666 but it's clear that feature bits are going to be distinct, so this is safe to do anyway. See lightning/bolts#680 Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This preempts the acceptance of lightning/bolts#666 but it's clear that feature bits are going to be distinct, so this is safe to do anyway. See lightning/bolts#680 Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Concept ACK, this is much easier to reason about 👍
| 4/5 | `option_upfront_shutdown_script` | Commits to a shutdown scriptpubkey when opening channel | IN | [BOLT #2][bolt02-open] | | ||
| 6/7 | `gossip_queries` | More sophisticated gossip control | IN | [BOLT #7][bolt07-query] | | ||
| 8/9 | `var_onion_optin` | Requires/supports variable-length routing onion payloads | IN | [Routing Onion Specification][bolt04] | | ||
| 10/11 | `gossip_queries_ex` | Gossip queries can include additional information | IN | [BOLT #7][bolt07-query] | |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
missing 12/13 for option_static_remote_key
? maybe just needs rebase?
01-messaging.md
Outdated
@@ -246,15 +246,13 @@ The receiving node: | |||
|
|||
#### Rationale | |||
|
|||
There used to be two feature bitfields here, but the first is now ignored. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
it's no longer backwards compatible to ignore the global features, don't they need to be merged with the existing localfeatures
? It's okay for writers to stop putting features there (and move everything to features
, but we can't retroactively stop parsing features that were placed in global but not local, e.g. option_var_onion
This preempts the acceptance of lightning/bolts#666 but it's clear that feature bits are going to be distinct, so this is safe to do anyway. See lightning/bolts#680 Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This preempts the acceptance of lightning/bolts#666 but it's clear that feature bits are going to be distinct, so this is safe to do anyway. See lightning/bolts#680 Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This preempts the acceptance of lightning/bolts#666 but it's clear that feature bits are going to be distinct, so this is safe to do anyway. See lightning/bolts#680 Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
|
||
The following `globalfeatures` bits are currently assigned by this specification: | ||
The Context column decodes as follows: | ||
* `I`: presented in the `init` message. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If this is I
, what will our letter for invoice feature bits be? Maybe P
for payment request?
Should we carve out the space for invoice feature bits as well in this PR?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
from the candidate features, we could start to specify that var_onion_option
should be in invoices
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think your first comment raises an interesting point to discuss: do we want to merge invoice feature bits with node feature bits? If we do #643 can't use bits 0-3 (which it currently does) since it clashes with node features.
I think it may be useful to merge invoice feature bits with node feature bits as it sounds more future-proof. The only downside I can see is that it uses more space in the invoice, so QR codes might end up being big. But we can probably compress the features bitfield quite efficiently if it becomes large but we only want to set a few bits in it. WDYT?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
at this point I think we should forbid colliding feature bits altogether, so all feature bit spaces should share the same namespace. however there are some node features that don’t make sense as invoice features, e.g. support for gossip sync protocols, and those don’t need to be present.
the space for routing hints will probably dwarf the feature bits in the invoice for some time, but agree that a more space efficient representation for feature bits is worth exploring. I’ve been looking into a combination of RLE and/or sparse encodings, tho the more difficult task will be signaling support for the newer format without parsing any feature bits, but we have options :P
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
tho the more difficult task will be signaling support for the newer format without parsing any feature bits, but we have options :P
Feature-bit-ception 💥
01-messaging.md
Outdated
* [`gflen*byte`:`globalfeatures`] | ||
* [`u16`:`lflen`] | ||
* [`lflen*byte`:`localfeatures`] | ||
* [`u16`:`ignorelen`] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Reminder to revert back to existing.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done
2003ec8
to
5b19dde
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looking good! I haven't implemented it in Eclair yet but it's going to be a fairly easy change.
01-messaging.md
Outdated
|
||
1. type: 16 (`init`) | ||
2. data: | ||
* [`u16`:`gflen`] | ||
* [`gflen*byte`:`globalfeatures`] | ||
* [`u16`:`lflen`] | ||
* [`lflen*byte`:`localfeatures`] | ||
* [`flen*byte`:`localfeatures`] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
And the line before should be:
* [`u16`:`flen`]
01-messaging.md
Outdated
|
||
#### Requirements | ||
|
||
The sending node: | ||
- MUST send `init` as the first Lightning message for any connection. | ||
- MUST set feature bits as defined in [BOLT #9](09-features.md). | ||
- MUST set any undefined feature bits to 0. | ||
- SHOULD use the minimum lengths required to represent the feature fields. | ||
- SHOULD NOT set features greater than 11 in `globalfeatures`. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It does feel safer indeed to use 13 here.
| 3 | `initial_routing_sync` | Sending node needs a complete routing information dump | I | [BOLT #7][bolt07-sync] | | ||
| 4/5 | `option_upfront_shutdown_script` | Commits to a shutdown scriptpubkey when opening channel | IN | [BOLT #2][bolt02-open] | | ||
| 6/7 | `gossip_queries` | More sophisticated gossip control | IN | [BOLT #7][bolt07-query] | | ||
| 8/9 | `var_onion_optin` | Requires/supports variable-length routing onion payloads | IN | [Routing Onion Specification][bolt04] | |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We're not using it either for p2p, it will currently only be useful in the Bolt 11 invoice or inferred by other feature bits (MPP, Trampoline, etc).
Nice yeah it's not too bad, been running the version above on my node for the last day or so w/o issue |
My feeling is that (almost?) everything should be in init msgs, since it's more current than node_announcement. Especially since there's no node_announcement if the node has no channels yet? |
Here's the same version, with commits cleaned up into 3 distinct commits and rebased onto master (which includes option_static_remotekey): this is the one I actually plan on merging: https://github.com/rustyrussell/lightning-rfc/commits/flat-features-flattened |
ACK, looks good! |
Note this is already implemented on c-lightning 0.7.3, so you can test via BLUEIRON (mainnet) or SLICKERMASTER (testnet): Mainnet: 024b9a1fa8e006f1e3937f65f66c408e6da8e1ca728ea43222a7381df1cc449605@128.199.202.168 Testnet: 031a3478d481b92e3c28810228252898c5f0d82fc4d07f5210c4f34d4aba56b769@165.227.30.200 |
Needs a rebase then can be merged at will! |
2092c10
to
ed1a398
Compare
We simply specify, in each case, where they will appear ("Context"). Because `globalfeatures` is already in use, we fold that into the renamed `localfeatures` field to unify them (now called `features`), but dissuade further use. Note also: we REQUIRE minimal `features` field in channel_announcement, since otherwise both sides of channel will not agree and not be able to create their signatures! Consider these theoretical future features: `opt_dlog_chan`: a new channel type which uses a new discrete log HTLC type, but can't support traditional HTLC: * `init`: presents as odd (optional) or even (if traditional channels not supported) * `node_announcement`: the same as above, so you can seek suitable peers. * `channel_announcement`: presents as even (compulsory), since users need to use the new HTLCs. `opt_wumbochan`: a node which allows channels > 2^24 satoshis: * `init`: presents as odd (optional), or maybe even (if you only want giant channels) * `node_announcement`: the same as above, so you can seek suitable peers. * `channel_announcement`: not present, since size of channel indicates capacity. `opt_wumbohtlc`: a channel which allows HTLCs > 2^32 millisatoshis: * `init`: presents as odd (optional), or even (compulsory) * `node_announcement`: the same as above, so you can seek suitable peers. * `channel_announcement`: odd (optional) since you can use the channel without understanding what this option means. Signed-off-by: Rusty Russell <rusty@rustcorp.com.au> Co-Authored-By: Bastien Teinturier <31281497+t-bast@users.noreply.github.com>
The feature fields refer to the properties of the channel/node, not the message itself, so we can still propagate them (and should, to avoid splitting the network). If we want to make an incompatible announcement message, we'll use a different type, or insert an even TLV type. Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
A bit less dense, but avoids a separate feature space. Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
ed1a398
to
0b7bbeb
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ACK 0b7bbeb
Implement lightning/bolts#666 We keep the global/local split in Commitments to avoid backwards incompatibility in the codec.
Implement lightning/bolts#666 We keep the global/local split in Commitments to avoid backwards incompatibility in the codec.
Implement lightning/bolts#666 We keep the global/local split in Commitments to avoid backwards incompatibility in the codec.
Implement lightning/bolts#666 We keep the global/local split in Commitments to avoid backwards incompatibility in the codec.
Implement lightning/bolts#666 We keep the global/local split in Commitments to avoid backwards incompatibility in the codec.
Implement lightning/bolts#666 Keep the global/local split in Commitments to avoid backwards incompatibility in the codec. Remove allowMultiPart API field: we instead rely on the MPP feature being set in nodeParams. That means MPP-enabled nodes need to update their reference.conf. Rework features: * Add types to allow cleaner dependency validation. * Most of the time we don't care whether a feature is activated as optional or mandatory, which caused duplicate code. This is now handled more cleanly. * It also paves the way to annotate features with the places they should be advertised (Init vs NodeAnn vs ChannelAnn vs invoice).
This implements: - flat feature bits lightning/bolts#666 - feature bit dependencies lightning/bolts#719
This implements: - flat feature bits lightning/bolts#666 - feature bit dependencies lightning/bolts#719
This implements: - flat feature bits lightning/bolts#666 - feature bit dependencies lightning/bolts#719
This implements: - flat feature bits lightning/bolts#666 - feature bit dependencies lightning/bolts#719
This implements: - flat feature bits lightning/bolts#666 - feature bit dependencies lightning/bolts#719
This implements: - flat feature bits lightning/bolts#666 - feature bit dependencies lightning/bolts#719
We simply specify how they're presented in the three different contexts.
Consider these theoretical future features:
opt_dlog_chan
: a new channel type which uses a new discrete log HTLCtype, but can't support traditional HTLC:
init
: presents as odd (optional) or even (if traditional channelsnot supported)
node_announcement
: the same as above, so you can seek suitable peers.channel_announcement
: presents as even (compulsory), since users needto use the new HTLCs.
opt_wumbochan
: a node which allows channels > 2^24 satoshis:init
: presents as odd (optional), or maybe even (if you only wantgiant channels)
node_announcement
: the same as above, so you can seek suitable peers.channel_announcement
: not present, since size of channel indicatescapacity.
opt_wumbohtlc
: a channel which allows HTLCs > 2^32 millisatoshis:init
: presents as odd (optional), or even (compulsory)node_announcement
: the same as above, so you can seek suitable peers.channel_announcement
: odd (optional) since you can use the channelwithout understanding what this option means.
Closes: #571