Broadcast known message optimization

[DasIch]

I hope it's ok, if I ask a question here. I'm wondering about the optimization introduced in this commit here: d96079d

I'm curious why this particular optimization was chosen. It seems to me there would be other ways of achieving this that are more obvious (to me at least) and would work better. Like acknowledging "gossip" messages with a "gossip_ok" messages and updating known when receiving a "gossip_ok" message. That would also allow for a couple of follow-up optimizations like not sending "gossip" messages, when all messages are known to the other node. You can further include messages in "gossip_ok" to gossip back.

Having said that I'm also new to distributed systems, so I suspect I might be missing something rather important and the above is a terrible idea for some reason. So I'd really appreciate to learn more about why you chose to optimize it in this way.

[jonhoo]

I talked about this a little bit during the stream when I wrote that commit, but since then I've come around to (again) thinking we should have an explicit ACK to gossip messages. The main reason isn't so much to avoid sending messages though, but rather that the gossip topology isn't necessary symmetrical. That is, A may gossip to B, but B may never gossip to A. As a result, in the current implementation, A will never learn which messages B has received, and will thus likely continue to re-send old messages to B.

I'll likely end up fixing this the next time I do a stream on this :)