My cluster separated into one cluster per node

[theikell]

I expect the cause of this was related to an Azure network issue, but how can I monitor or better yet, prevent this from happening again.

Here's my story. I'm running development cluster of 3 VM's on Azure, they are all members of the same VLAN. All are running ES 1.2.2. I've setup discovery as per the docs. Everything works great and has been for about a month. Yesterday I started experiencing problems. While investigating I remoted into each node and logged onto Marvel using localhost. Each node reported that they were the only node in the cluster and that they were the master.

I restarted the ES service on two of the nodes and upon restart they discovered node that I did not restart, and the cluster reformed as expected.

I suspicion is that a network problem caused a communication failure between the nodes but when they network problem was resolved the nodes did not re-discover each other. I assume the discovery process only happens when a node starts, true?

I expect there is nothing I can do to prevent this from happening since it seems to have been initiated by an issue in the Azure datacenter, but is there a way to monitor this, or have the nodes attempt to rediscover each other with out me having to manually restart them?

[dadoonet]

I agree. I think something is not working as expected in that case.
I need to investigate and try to reproduce.

BTW, I think you should set minimum_master_nodes:2 in your case to avoid the split brain issue.

[theikell]

Thank you for your advice regarding minimum_master_nodes, I'm making that change now.