Use etcd for atomic broadcast

[jabolina]

Proposal

Since the main focus for this primitive is to be reliable, the goal should be to use a atomic broadcast implementation and remove the current RabbitMQ usage. If possible, the final goal should be to implement a generic broadcast primitive and remove any dependency for libraries.

etcd

When using etcd, a first setup must be executed, pretty much what already happens when using the RabbitMQ version. The relt implementation will change to be a etcd client, where commands will be applied upon a KV store, and peers can use the watch API to listen for changes applied on the KV by other peers using the put API.

Starting peers

When starting a new peer, the configuration is something similar with the current RabbitMQ impl. The peer will contain an address, that should be the address to the etcd server, the other peers will also connect to the given etcd server, so there is no need for manually creating a client discovery.

The peer will receive a partition name, that will work as an alias for receiving messages. The peer will listen for messages on this partition, which means that when sending a message, the destination is a partition name, not a peer address. Internally, a peer will be watching for changes on the KV store, the key will be the partition name and when sending a message means writing on the KV store. For developing something more reliable, the projects displayed on the etcd page will be used and listed here for reference.

Using this approach, we must have a primitive that totally orders and delivers the message for the destination peers. Any update during development will be listed here and any documentation as well.

---

References:

Metafora - https://github.com/lytics/metafora
https://www.compose.com/articles/utilizing-etcd3-with-go/