Minority failure during cluster configuration change risks deadlock

[andres-erbsen]

Adding a node to a degraded cluster and then electing it as the leader causes the addition not to reach the nodes that were down. In particular, starting with a cluster of 3 nodes a, b, c and adding d while a is not available and then electing d as the leader makes a unable to participate in the cluster. a will remain useless as long as d is the leader. This is especially bad in case another node (e.g. c) goes down, and the cluster would have to rely on the participation of a to make progress. Thus it is possible to have a situation where there is a live majority from both the old cluster (a, b) and the new cluster (a, b, d) but the cluster is stuck because a does not listen to d even though d is the leader.

I am not sure how to fix this. The invariant from the thesis does not seem applicable since etcd configuration changes apply at a different time and I do not understand the correctness reasoning behind the etcd configuration change algorithm well enough to make changes to it.

A screencast of me reproducing this issue is available at
http://web.mit.edu/andreser/Public/etcd-reconfiguration-deadlock/index.html

[yichengq]

@andres-erbsen I could reproduce, so this is a bug.

The reason AFAIK is that a doesn't know the address of d, so it cannot send raft message to it. We need to let a know d's address in some way.

[xiang90]

@andres-erbsen Interesting... This is a bug. As far as I can see, the root cause is not raft but a bug in transportation layer. We will fix it soon.

[daniel-ziegler]

cc @aphyr

[daniel-ziegler]

Any updates on this?

[xiang90]

@daniel-ziegler I am going to look into this soon. This was not a priority since it does not happen frequently in real world.