[WIP] BOLT 7: Inventory-based gossip #584

sstone · 2019-02-28T15:02:45Z

This is a proposal for adding inventory messages to the gossip protocol. With the current design, nodes which receive gossip messages (node announcements, channel announcements and updates) will relay them to their peers, which will result in duplicates for nodes that are connected to many different peers.

Nodes which support the option_inv_gossip feature will instead broadcast inventory messages which contain identifiers for gossip messages they've received. Receiving node will compare these identifiers to their local view of the routing table, and ask for missing or outdated messages using channel queries. This implies that option_inv_gossip cannot be used without gossip_queries.

It builds upon PR #571 (unification of feature bits, which includes a definition for option_inv_gossip) and #557 (extended channel queries).
More specifically, it is a "companion" PR to #557:

extended channel queries are used to efficiently synchronise routing tables between a node that is often offline and a very limited number of peers
inventory-based gossip is used to minimize duplicate gossip traffic for nodes that are connected to many different peers
they share similar concern: how to efficiently advertise information about channel announcement and updates

In fact, the inventory message that I propose to use here is almost identical to an extended reply_channel_range message and includes short channel ids, checksums and timestamps, to allow receiving to efficiently query messages based on content (checksum) and timestamps.

Node announcements have been left out on purpose because they have a much more limited impact on bandwidth than channel updates, and because gains would be smaller (you would still need to advertise node ids which are 33 bytes long).

rustyrussell

I'm going to push a small formatting fixup (we generate from the spec, so this makes it much easier to implement!)

rustyrussell · 2019-03-18T09:40:45Z

07-routing-gossip.md

 ### The `query_short_channel_ids`/`reply_short_channel_ids_end` Messages

 1. type: 261 (`query_short_channel_ids`) (`gossip_queries`)
 2. data:
    * [`32`:`chain_hash`]
    * [`2`:`len`]
    * [`len`:`encoded_short_ids`]
+    * [`2`:`option_len`]
+    * [`len`:`option_encoded_query_flags`]


len needs to be option_len here.

rustyrussell · 2019-03-18T09:42:21Z

07-routing-gossip.md

@@ -636,6 +656,9 @@ timeouts.  It also causes a natural ratelimiting of queries.
    * [`1`:`complete`]
    * [`2`:`len`]
    * [`len`:`encoded_short_ids`]
+    * [`1`:`option_extended_query_flag`]
+    * [`2`:`option_extended_info_len`]
+    * [`len`:`option_extended_info`]


Here too, len -> option_extended_info_len.

n1bor · 2019-03-20T20:17:28Z

Other thought was to get nodes to opt-out of updates for most channels from most peers. Since node has a full map of the network it could tell most peers not to send updates for a list of channels that it knows it will get quicker from multiple other peers.
Some info here: ACINQ/eclair#864

pm47 · 2019-03-20T20:33:28Z

Fwiw we can already opt out entirely of updates on a peer by peer basis by not sending `gossip_timestamp_filter`. Le mer. 20 mars 2019 à 21:17, n1bor <notifications@github.com> a écrit :

…

Other thought was to get nodes to opt-out of updates for most channels from most peers. Since node has a full map of the network it could tell most peers not to send updates for a list of channels that it knows it will get quicker from multiple other peers. Some info here: ACINQ/eclair#864 <ACINQ/eclair#864> — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#584 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AB7yvk3Hfrnfn4lYSv9XNsNlAhmDEBgGks5vYpdbgaJpZM4bXBIw> .

Roasbeef · 2019-05-01T02:57:56Z

07-routing-gossip.md

+
+A more efficient scheme based on inventory messages, similar to how transactions and blocks are propagated on the bitcoin p2p network, can optionally be used: if the `option_inv_gossip` feature bit is set, instead of broadcasting `channel_annoucement` and `channel_update` messages, nodes will broadcast a `gossip_inventory` message which is similar to a `reply_channel_range` message and can be used to filter announcements by timestamp or by content (checksum):
+
+1. type: 266 (`gossip_inventory`)


Rather than a single message for all inventory types I think we should instead have distinct messages as this allows us to open encode certain fields that are relevant if a node wants to filter out a class of advertisements. For example a channel_inv message would have things like the size of the channel, or the nodes that it's connecting included. This would let nodes performing filtering such as ignoring all channels that are below a certain size. With the checksum approach the node has no idea what it's about to fetch, and will waste bandwidth by fetching something that it will instantly abandon.

While we're at it, we may also need to extend the prior queries in order to allow fetching a node announcement.

I've rebased to get the latest changes from #557, and I'm using TLV encoding now. I've adding a node inventory messages (just a list of node ids).
I don't think that we need to include channel capacity in the inv message: you would have to trust the sender anyway, and tiny channels will progressively go away.
I don't think we should include the node ids it's connecting: how would you use it ? You don't know what a new channel enables from a routing pov until you have the rest of the graph

n1bor · 2019-05-01T07:30:11Z

Just some stats on scale of issues:
https://lightningconductor.net/grafana/d/CZQF3HzWz/lightning-stats?orgId=1&from=now-7d&to=now&fullscreen&panelId=22
Is for node:
03c436af41160a355fc1ed230a64f6a64bcbd2ae50f12171d1318f9782602be601@mainnet.lightningconductor.net:9735

Roasbeef · 2019-05-01T08:04:38Z

What implementation and version is that node running? I see about an order of magnitude less traffic on my node personally.

…

On Wed, May 1, 2019, 12:30 AM n1bor ***@***.***> wrote: Just some stats on scale of issues: https://lightningconductor.net/grafana/d/CZQF3HzWz/lightning-stats?orgId=1&from=now-7d&to=now&fullscreen&panelId=22 Is for node: ***@***.*** :9735 — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#584 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAHTWLWGAJMV26MS5GOBA23PTFBILANCNFSM4G24CIYA> .

n1bor · 2019-05-01T18:16:02Z

A 2 week old version of Eclair.
ACINQ/eclair@9032da5
They did commit an optimisation a week ago. Will move onto that in a few day.
I think my stats are good as I am just scraping the log file with logstat. And I did double check a few of the peeks and they were real.
Am average 5.5k updates a minute... That grafana site has number of peers - is about 180.
https://lightningconductor.net/grafana/d/CZQF3HzWz/lightning-stats?fullscreen&panelId=14

pm47 · 2019-05-01T19:21:21Z

Those are orthogonal things: this PR reduces the bandwidth used by gossip, all things remaining equal (even if the number of channel_updates remained the same).

@n1bor seems to have a pathological case of flapping channels, which caused eclair to generate a constant stream of channel_updates. That's an implementation problem, not a spec one, and our recent fix (ACINQ/eclair@fb84dfb) alleviates this issue.

n1bor · 2019-05-01T21:52:00Z

Updated to HEAD de5a782 - lets see if it settles down in the next few hours.

UPDATE---
Just to be clear the 5k updates per minute are the total number of updates (mainly relays) sent by my node.
It has about 180 peers so is about 30 updates sent to each peer each minute.

So this change would reduce size of each update from about 140bytes to about 13 - and these are both sent and received. But probably as importantly would reduce the huge number of small network packets we currently send/receive.

rustyrussell · 2019-06-14T06:35:17Z

OK, I'm working on TLV parsing, so some cleanups (and introduce some conventions). Nothing significant changed in the format, except removal of some redundant lengths inside TLV records.

1. Uses 'tlvs' for the "length" of tlv streams. 2. Allow - inside length of fields, since it's useful given an implied tlv_len. Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

cfromknecht · 2019-07-18T23:21:16Z

07-routing-gossip.md

+
+`channel_inv` contains information about channel announcements and updates. It uses the exact same encoding as `reply_channel_range` and includes:
+ - a list of `short_channel_id`s
+ - an optional `timestamps`, which includes `channel_update` timestamps


is it possible to avoid the redundancy of sending timestamps for both edges every time?

Not atm. You can choose to not send timestamps at all, but if you do you must include timestamps for all updates. I'm not sure how sending some but not all of them could be used ?

cfromknecht · 2019-07-18T23:21:18Z

07-routing-gossip.md

+        * [`tlv_len`:`encoded_checksums`]
+   1. type: 7 (`node_inv`)
+   2. data:
+      * [`tlv_len`:`node_ids`]


from my understanding, #557 doesn't have a way of querying for node_announcement without the corresponding short_channel_id. Can you explain how one would retrieve the short_channel_id from this message to use in the query? If node_ids is a different size from encoded_short_ids, which one should the receiver use?

That's a good point and now I see why @Roasbeef wanted a different INV message for node announcements. A simple answer would be to include node_id, timestamp and short_channel_id in the INV message, receiver can then use the provided short_channel_id to query the node announcement.

07-routing-gossip.md

TheBlueMatt · 2020-03-06T01:53:28Z

This needs a rebase. If we're gonna add more stuff to the route-sync state machine, can we improve it? The current design is super complicated, and its just kinda grown organically without a lot of thought to the end state. It has grown based on whats available without a lot of care to whats allowed (eg it relies heavily on sequence id fields being timestamps, which the spec doesn't even require, not to mention stale updates my propagate late and uses zlib as a cheap way to communicate data, instead of a purpose-designed thing like minisketch). At various points, I've suggested we move towards where Bitcoin Core is going and:

Start with an initial sync based on last timestamp fields with a two hour buffer (we need to constrain the timestamp fields to be at least after block header timestamps, but those are only +/- 2 hours, so we can't assume more than that).
Use minisketch (or just an inv list, if we don't want to take the dependency) to figure out what you need that was missed maybe because it wasn't relayed in a timely fashion or because its somewhere in the two hour buffer,
Use invs thereafter.

This would remove a ton of existing complexity.

Inventory messages is an optional feature that is used to reduce duplicate gossip traffic. Instead of sending `channel_annoucement` or `channel_update` messages, nodes will send an inventory message to peers that support this option. Inventory messages are similar to channel queries and contain a list of short channel ids, checksums and timestamps. Receiving nodes will check inventory data and use channel queries to retrieve missing or outdated channel announcements and updates.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

We add a specific node inventory message that includes node id, timestamp, and short channel id. The short channel id will be used as an alias to query the corresponding announcement, if needed, using extended channel range queries.

sstone added the 2019-03-04 label Mar 2, 2019

rustyrussell removed the 2019-03-04 label Mar 18, 2019

rustyrussell reviewed Mar 18, 2019

View reviewed changes

rustyrussell self-requested a review March 18, 2019 10:06

Roasbeef requested changes May 1, 2019

View reviewed changes

n1bor mentioned this pull request May 2, 2019

Flapping Channels - is this the cause of 5k update per minute? ACINQ/eclair#978

Closed

sstone force-pushed the bolt7-inv-gossip branch from b39997c to aa871f1 Compare May 10, 2019 16:10

sstone force-pushed the bolt7-inv-gossip branch from 56412f9 to 88ed789 Compare July 18, 2019 09:55

sstone mentioned this pull request Jul 18, 2019

BOLT07: Inventory Gossip #647

Open

cfromknecht reviewed Jul 18, 2019

View reviewed changes

TheBlueMatt mentioned this pull request Mar 6, 2020

Swap announcement/update serials for block timestamp. lightningdevkit/rust-lightning#493

Closed

sstone and others added 5 commits March 13, 2020 17:43

BOLT 7: fix for b7df807

8e4e381

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

Use TLV encoding

4d95394

Fixup: make INV message consistent with extended channel queries

be28fc5

Create a specific node inventory message

4e68461

We add a specific node inventory message that includes node id, timestamp, and short channel id. The short channel id will be used as an alias to query the corresponding announcement, if needed, using extended channel range queries.

sstone force-pushed the bolt7-inv-gossip branch from 2f41dd4 to 4e68461 Compare March 13, 2020 16:47

TheBlueMatt mentioned this pull request May 3, 2020

Channel Graph Fetching from newer lightning nodes lightningdevkit/rust-lightning#615

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WIP] BOLT 7: Inventory-based gossip #584

[WIP] BOLT 7: Inventory-based gossip #584

sstone commented Feb 28, 2019

rustyrussell left a comment

rustyrussell Mar 18, 2019

rustyrussell Mar 18, 2019

n1bor commented Mar 20, 2019

pm47 commented Mar 20, 2019 via email

Roasbeef May 1, 2019

sstone May 10, 2019

n1bor commented May 1, 2019

Roasbeef commented May 1, 2019 via email

n1bor commented May 1, 2019 •

edited

Loading

pm47 commented May 1, 2019

n1bor commented May 1, 2019 •

edited

Loading

rustyrussell commented Jun 14, 2019

cfromknecht Jul 18, 2019

sstone Jul 19, 2019

cfromknecht Jul 18, 2019

sstone Jul 19, 2019

TheBlueMatt commented Mar 6, 2020


		A more efficient scheme based on inventory messages, similar to how transactions and blocks are propagated on the bitcoin p2p network, can optionally be used: if the `option_inv_gossip` feature bit is set, instead of broadcasting `channel_annoucement` and `channel_update` messages, nodes will broadcast a `gossip_inventory` message which is similar to a `reply_channel_range` message and can be used to filter announcements by timestamp or by content (checksum):

		1. type: 266 (`gossip_inventory`)

[WIP] BOLT 7: Inventory-based gossip #584

Are you sure you want to change the base?

[WIP] BOLT 7: Inventory-based gossip #584

Conversation

sstone commented Feb 28, 2019

rustyrussell left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

n1bor commented Mar 20, 2019

pm47 commented Mar 20, 2019 via email

Choose a reason for hiding this comment

Choose a reason for hiding this comment

n1bor commented May 1, 2019

Roasbeef commented May 1, 2019 via email

n1bor commented May 1, 2019 • edited Loading

pm47 commented May 1, 2019

n1bor commented May 1, 2019 • edited Loading

rustyrussell commented Jun 14, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

TheBlueMatt commented Mar 6, 2020

n1bor commented May 1, 2019 •

edited

Loading

n1bor commented May 1, 2019 •

edited

Loading