Restore process can restore incompatible minimum_master_nodes setting

[grantr]

If I have 3 masters and accidentally set minimum_master_nodes to 4 dynamically, the cluster will stop. While the cluster is stopped, there's no way to update settings, so I have to do a full restart.

If I updated a persistent setting, then not even a full restart will fix the issue. I have to manually edit cluster state.

You could get out of this situation by adding master nodes until the new minimum is reached, but that's not a complete solution because I might have fat fingered minimum_master_nodes to 100 or something like that.

I think it'd be worth adding some validation to ensure that an update to minimum_master_nodes won't accidentally put the cluster in an unrecoverable state.

[bleskes]

Agreed it's a pain. This is already fixed in #8321 , which will be released with 1.5.0

I'm closing this, but if you feel something is missing from that PR, please feel free to re-open.

[grantr]

Excellent. Thanks much @bleskes!

[grantr]

@bleskes it just occurred to me that restoring global state from a snapshot can update persistent settings:

http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-snapshots.html#_restore

The restored persistent settings are added to the existing persistent settings.

@imotov what happens if a snapshot restores a persistent setting for minimum_master_nodes that is greater than the current master count?

[imotov]

@grantr yes, this indeed can be an issue. Thanks!

[grantr]

Thanks for fixing this @imotov!