Spike: Use raft consensus for networking

[ch1bo]

Why

We created a new test suite about resilience of our network stack in #1532 (see also #1106, #1436, #1505). With this in place, we can now explore various means to reach our goal of a crash-tolerant network layer.

This fairly old research paper explored various consensus protocols used in blockchain space and reminds us of the correspondence between consensus and broadcasts:

the form of consensus relevant for blockchain is technically known as atomic
broadcast

Furthermore, it listed at least one of these early, permissioned blockchains that achieved crash-tolerance of $t < n/2$ by simply using etcd with its Raft consensus algorithm.

What

• Run fault injection tests with a hydra-node network connected through etcd
• Create a PR with results of the spike, but do not merge it

How

• Create a Hydra.Network.Etcd network component that implements broadcast using the replicated log of etcd
  • Move authentication into application
  • Re-use --peer from command line
  • Fork etcd when starting
  • Message type + sender as key
  • Use watch and revisions to be notified of messages while offline
• Maybe: Compact messages (this will upper bound reliance to faults?) or use leases