Add implicit discovery.zen.maximum_master_nodes

[pickypg]

We already have support for maintaining quorum of the minimum number of master nodes. It may also be nice to have an inverse setting to limit the expected number of master nodes that can join the cluster.

This is purely to solve operational misconfigurations, but it can also be implicitly determined by solving for M (eligible master nodes) from the quorum equation given discovery.zen.minimum_master_nodes.

quorum = (M / 2) + 1
M = 2 * (quorum - 1)

We can use this to prevent misconfigurations from becoming a problem by failing for the same reason when quorum is not met.

quorum <= running masters <=  M

rather than just

quorum <= running masters

This also makes me wonder if we should deprecate discovery.zen.minimum_master_nodes in favor of a discovery.zen.expected_master_nodes (defaults to 1), similar to gateway.expected_master_nodes. Then we can calculate quorum ourselves to avoid confusion (truncation/integer division can sometimes catch people off guard).

[clintongormley]

Hmmm, I'm against this for a few reasons:

• having too many master nodes is not a problem, why add yet another setting
• explaining that the number of available masters is less than minimum_master_nodes is easier than explaining that it is less than 1+int(expected/2)
• minimum_master_nodes is widely publicised, so we would break blogs everywhere
• it is possible that we will be able to remove this setting completely in the future, with changes to Zen