cluster broken after switches upgrade

[jbdamiano]

Hello,

We have actually 7 nodes. We use elasticsearch 1.3.2.

We are configured in unicast mode with a minimum consistency of 4
Two days ago, in the night, network team has done a switches firmware upgrade of all production switches.

The result of this upgrade has been catastrophic for the cluster.
The cluster has been split in two parts with some node in first and second part

Node 3 and Node 5 was master together

Node 3 with Node 1, Node3 and Node 6
Node 5 with Node 1, Node2, Node4, Node 5, Node 6 and Node 7

Fortunately we have a second cluster on other lab which the same data and we have been able to do a snapshot/restore from this cluster to the first one after a rolling restart.

And don't understand how a switch upgrade, and a connectivity lost, can result of a corrupted cluster allocation.

Regards,

[bleskes]

@GOUKETSU that's very unfortunate. This happens because of #2488 which was fixed with 1.4.0 . I suggest you upgrade as soon as you can.

I'm closing this as duplicate. Feel free to comment if you have any question.