feat: iroh-gossip

[Frando]

Description

This PR adds iroh-gossip. See n0-computer/iroh-sync#1 for the prehistory.

iroh-gossip exposes a pubsub-like API to broadcast messages on topics. Peers join a topic by connecting to at least one other peer. They can then broadcast messages to all other peers who joined a topic, and subscribe to messages. The protocol is based on HyParView, which gives us an overlay network of all nodes that joined a topic; and PlumTree to create an epidemic broadcast tree for reliably broadcasting messages to all online peers.

The new crate is structured as follows:

• The protocol implementation (mod proto) as an IO-less state machine. It exposes a single method to handle InEvents and returns an iterator of OutEvents which must be handled by the implementor (register timers, send messages, emit events). The implementation is generic over a PeerAddress trait, which should be an identifier for a peer. In our case, it will be a PeerId.

  • All protocol messages are namespaced by a TopicId, a 32 byte identifier. Topics are seperate swarms and broadcast scopes. The HyParView and PlumTree algorithms both work in the scope of a single topic. Thus, joining multiple topics increases the number of open connections to peers and the size of the local routing table.
  • The high level protocol lives in state::State. It mostly dispatches messages based on the topic ID into the matching topic::State. This is the actual protocol state for a topic. It, again, mostly dispatches messages onto the actual protocols: hyparview::State and plumtree::State.
  • The implementations of these two modules quite strictly follows their papers 1 and 2. If you find any deviations that are not documented, please report, this is likely a bug.

• The net module provides networking for the IO-less protocol impl, through the MagicEndpoint from iroh-net, with a (I think) quite straightforward actor implementation.

The net module has a smoke test, and the PR also includes a chat example that adds signed messages on the trustless base layer.

What this mostly needs is, I think, more testing and validation of the protocol implementation. For this to be practical, I think we should formulate some base usecase scenarios.

The proto module has a very simple simulator and some tests that can take environment variables to increase the cluster size. They passed with 10.000 nodes. They also report some stats. The numbers look OK-ish, but I'd love to compare them properly with either other implementations or estimations from the original paper or other resoures. However, those are hard to find.

Notes & open questions

• Right now the plumtree module builds an unbounded cache of all broadcast messages. This will have to change. The simplest might be to move to an LRU cache for now. In the future, we might want to expose a method for the module to request expired messages from a backing storage or so. (The module needs to have a cache of messages to reply for requests by other peers. However these will only come within a reasonable amount of time after message publication / reception.) this is now implemented with an expiring cache
• Same for the list of received message IDs - they're also unbounded atm this is now implemented with an expiring cache
• We should validate the numbers - either by more simulation tests or by actually comparing them to the paper. However I couldn't find any simulation code from the papers.

Change checklist

• Self-review.
• Documentation updates if relevant.
• [ ]Tests if relevant. some tests are there, some more might be good

[dignifiedquire]

missing unit tests, ideally some proptests for the pieces in this file

[Frando]

I added basic unit tests for IndexSet, TimerMap and the new base32 utils.
What parts would you want to proptest? Can do, just not super sure if/where it would make sense here.