Trickle Gossip Messages

[cfromknecht]

LND currently constructs a new batch of gossip messages to broadcast every 90s using a trickleTimer in the gossiper. Every time the timer expires, the messages are deduplicated and immediately broadcast to all peers that are receiving updates.

This approach creates needlessly bursty resource usage on both the sending and receiving side, and profiling confirms that this creates punctuated equilibriums in terms of bandwidth/cpu/memory. We can achieve more stable and consistent resource usage by dispersing each newly created batch over the subsequent 90s instead of delivering them all at once.

Steps to completion:

1. Decide how often the sub-batches should be sent, e.g. 5 seconds
2. Split an announcement batch into the sub-batches, where the size of each sub-batch should be roughly len(announcementBatch)/numSubBatchesIn90s, bonus points for using slice tricks
3. Instead of queueing the entire announcmentBatch immediately, spawn a goroutine to perform the remainder of the queueing logic for each sub-batch on the chosen interval. Note that the set of GossipSyncers can change over this period, so the syncers and filtering logic should be evaluated at the time each sub-batch is processed

Additional Considerations:

• Should we enforce a minimum batch size? If we only have 10 updates to send, do we really want to space each message by 5 seconds?

[orbitalturtle]

Taking a stab at this

[johng]

@orbitalturtle apologies, should have messaged before I started on this

[orbitalturtle]

@johng np :)

[halseth]

One problem I see with this approach is that we risk broadcasting outdated messages if an update comes in before we are able to send out the sub-batch containing it.

@cfromknecht What about adding all messages to a "delay queue", where they are required to stay for trickleDelay time before they are broadcast. This queue can take care of de-duping messages, moving them to the back of the queue if they are updated. This way we can easily rate limit how much we read from this queue, to control the flow of messages being broadcast.

[cfromknecht]

@halseth i don't see that as an issue. by the time the subbatches are created, we've done all of the deduping required of us in interval i. if a newer message comes in before the batch goes out, how is it different than if we had sent the entire batch immediately and then send the newer one in the next batch?

Doing cross-interval cut through as you suggest is a good optimization, but i don't think it should prevent us from getting the simple approach in

[cfromknecht]

What about adding all messages to a "delay queue", where they are required to stay for trickleDelay time before they are broadcast.

Isn't this already what we do with the trickle delay timer? Seems we'd be doing this twice?

[halseth]

You are right, any new update will go through the same delay, so order will be maintained ofc :)

Cutting through the new updates probably won't make that much of a difference (the biggest win would maybe be to never broadcast messages from nodes making updates with less than 90s in between)