Cross AZ / Cross Region Deployments

[petermetz]

Description

As an operator of validator nodes I want to be able to invite other organizations to join my private cluster so that we can make our resources work together.

For example

Company A wants to build a public BIF deployment to back a paid service where people can transfer assets between 2 different networks.

Business is booming and soon Company A leadership decides that they want to partner with Company B so that they share the cost of running the infrastructure and double the marketing reach through their combined brands.

They issue a fancy press release announcing the partnership, shake hands and then give the problem of figuring out the rest to the engineering department who soon realizes that the data centers of Company B are on a different continent and moving their infrastructure is (of course) not an option.
So, they have to make it so that a cluster of BIF validator nodes are working together in a productive fashion even though the latency between them can be > 200ms at times which is a challenge when you have a leader election algorithm (Etcd/Raft) that is sensitive to latency.

Acceptance Criteria

1. Verify that we are able to tune Etcd so that cross AZ/R deployments are operational. Start here: https://github.com/etcd-io/etcd/blob/master/Documentation/tuning.md
2. If Etcd doesn't work, document very thoroughly what tests were done and what were the exact results based on which the conclusions were drawn.
3. Look into CockroachDB for inspiration on how they solve the exact same challenge in the context of their software. Document the learnings from there and include it in the architectural documentation of BIF regardless of what decision is made in the end of the story.

[petermetz]

Fixed by #123 (We no longer use or depend on etcd)