Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Restore process can restore incompatible `minimum_master_nodes` setting #8830

Closed
grantr opened this issue Dec 8, 2014 · 5 comments · Fixed by #9051

Comments

@grantr
Copy link

commented Dec 8, 2014

If I have 3 masters and accidentally set minimum_master_nodes to 4 dynamically, the cluster will stop. While the cluster is stopped, there's no way to update settings, so I have to do a full restart.

If I updated a persistent setting, then not even a full restart will fix the issue. I have to manually edit cluster state.

You could get out of this situation by adding master nodes until the new minimum is reached, but that's not a complete solution because I might have fat fingered minimum_master_nodes to 100 or something like that.

I think it'd be worth adding some validation to ensure that an update to minimum_master_nodes won't accidentally put the cluster in an unrecoverable state.

@bleskes

This comment has been minimized.

Copy link
Member

commented Dec 8, 2014

Agreed it's a pain. This is already fixed in #8321 , which will be released with 1.5.0

I'm closing this, but if you feel something is missing from that PR, please feel free to re-open.

@bleskes bleskes closed this Dec 8, 2014
@grantr

This comment has been minimized.

Copy link
Author

commented Dec 8, 2014

Excellent. Thanks much @bleskes!

@grantr

This comment has been minimized.

Copy link
Author

commented Dec 10, 2014

@bleskes it just occurred to me that restoring global state from a snapshot can update persistent settings:

http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-snapshots.html#_restore

The restored persistent settings are added to the existing persistent settings.

@imotov what happens if a snapshot restores a persistent setting for minimum_master_nodes that is greater than the current master count?

@imotov

This comment has been minimized.

Copy link
Member

commented Dec 24, 2014

@grantr yes, this indeed can be an issue. Thanks!

@imotov imotov reopened this Dec 24, 2014
@imotov imotov self-assigned this Dec 24, 2014
@imotov imotov changed the title Setting `minimum_master_nodes` incorrectly can stop the cluster Restore process can restore incompatible `minimum_master_nodes` setting Dec 24, 2014
@grantr

This comment has been minimized.

Copy link
Author

commented Dec 24, 2014

Thanks for fixing this @imotov!

imotov added a commit to imotov/elasticsearch that referenced this issue Jan 18, 2015
@imotov imotov closed this in #9051 Jan 18, 2015
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.