consul-replicate Many-Many DC replication

[deepagargit]

Hello,

I have a KV namespace /shared which is present in all N (say 3 for now) DCs, DC1, DC2, DC3.

The objective is to keep /shared in-sync and KV could be added in any DC and it should get sync to other DCs but optimal sync is desired.

Please address below 4 queries:

1. Does this mean (N*N) complexity ? Every DC try to pull replicate from "EACH" (N-1) other DCs

2. Is there a recommended topology design for N DCs to reduce the complexity to N of sync data within /shared KV namespace?

   Example :
   If DC1 adds KV, (N-1) DCs should pull from DC1 i.e. (N-1) pulls and (N-1) DCs should not try to pull same KV update from other DCs?
   When DC3 can get KV from DC1 , DC3 should not pull KV from DC2 even-though KV is added in DC2 /shared namespace and DC3 is watching DC2/shared for replicate

3. Will Cassandra or other DB suit well for such replicate between DCs to prevent N*N replication with /shared namespace?

Case-1:

If a KV is added in DC1 -

• DC2 consul-replica would replicate in DC2\shared from DC1
• DC3 consul-replica would replicate in DC3\shared from DC1

Now since KV is replicated in DC2\shared

• DC1 consul-replica would try replicate in DC1\shared from DC2 [Already present no action] >>>>>>>>>>> why does DC1 try to pull ?
• DC3 consul-replica would try replicate in DC3\shared from DC2 >>>>>>>>>>> why this replicate again it is already done via DC1 ?

Now since KV is replicated in DC3\shared

• DC1 consul-replica would try to replicate in DC1\shared from DC3 [Already present no action] >>>>>>>>>>> why does DC1 try to pull ?
• DC2 consul-replica would try to replicate in DC2\shared from DC3 >>>>>>>>>>> why this replicate again it is already done via DC1 ?

Case-2:

If a K1-V1 is added in DC1

• DC2 consul-replica would replicate in DC2\shared

Now since K1-V1 is replicated in DC2\shared

• DC1 consul-replicate would try to get replicate in DC1\shared

Now suppose K1-V2 is updated in DC1 in meantime

Meanwhile DC1 gets K1-V1 from DC2,

1. Will DC1 hold K1-V2 or K1-V1 (from DC2) based on time-stamp?
   Will Casanndra suit better to decide based on time-stamp?

Case 3:

If DC2 is running consul-replicate pull from DC1
DC1 has 3 servers
if KV is added in DC1

1. Will DC2 try to pull only one copy of KV from DC1 or pull one copy per server in DC1 i.e. 3 copy of same KV from DC1
   I assume it is leader based pull from DC1 and DC2 will pull only one copy of KV from DC1 consul-replicate server leader

2. If DC1 has 3 servers and has say server-1 as consul leader
   Will the consul-replicate server leader in DC1 be always same as consul leader say server-1
   or consul-replicate leader election within DC1 servers would be independent of consul leader in DC1 servers?

   consul-replicate server leader in DC1 : sends the replicate data to DC2 on pull from DC2

Thanks,
Deepak

[armon]

The design of consul-replicate is such that it is only suited for master/slave replication.
This means a single datacenter should be considered the source of truth and the “master”,
from which all other DCs replicate keys.

As a result, the replication complexity is only N, since each slave is directly replicating from
the master. This means you get a “star” topology, with the master in the center.

Each datacenter uses an elected leader to serialize writes so there is never a multi-version
conflict. Because there is no coordination between datacenters for writes, there is no way
to reconcile conflicting writes. If key “test” has value “foo” in DC1 and “bar” in DC2 there
is no way for active-active replication to determine the correct value without substantial
changes to Consul.