Advanced Cluster Management Features

[gingerwizard]

Proposal to support:

1. Full Cluster Stop cluster
2. Stop/Start/Restart Specific Node
3. Full Cluster Restart
4. Rolling Restart

Some of the above could already be achieved. It may not be appropriate to include this functionality in the default role - we may wish to create a separate management role.

To be discussed.

[soenkeliebau]

I would indeed suggest splitting management features into an extra role or even playbook. I have in the past usually created specific playbooks for a restart and then called that from a shell script to facilitate usage.

I have pushed my rolling restart script to https://github.com/opencore/es_rollingrestart please feel free to reuse anything that might be useful.
This specific version has undergone some changes and not been thoroughly tested yet, if you notice any strange behavior or errors please let me know!

[jpcarey]

@gingerwizard I created a similar 1.x rolling restart in the past, adding it to the task directory of the role. You could conditionally use include in the main playbook to trigger such a task. To me this makes more sense, having administrative tasks stored in the appropriate roles - but never called/included unless someone did so explicitly. I worked in a large organization where we had over 200 roles, and adding "extra" misc roles would just add clutter and confusion. Roles are generic, which would include their child tasks; a playbook should control how the role is used.

I wouldn't mind rewriting a rolling restart for 2.x, and adding in more protection (check minimum_master_nodes before running, fail if cluster ever goes red, etc). I hate the ansible uri module, but I didn't want to spend time to write it as a plugin or module.

[gingerwizard]

@jpcarey i think we should provide a separate role to do this. Anything in this role needs to be tested. Testing the functionality above will be tricky i suspect and add a maintenance cost that is likely to be unsustainable moving forward. This is also not supported in the puppet module - we are trying to aim for consistent behaviour here.

@jakommo @dliappis for input.

[jakommo]

I'm +1 for a dedicated role for these tasks

[strootman]

+1 for separating.

Shoving too much stuff into a single role is a bad anti-pattern in ansible and ansible-galaxy. Makes re-usability take a dive.

[barryib]

@gingerwizard @jpcarey I think this duplicate with #126?

[gingerwizard]

yes duplicate and agreed this will not be added. Thanks all.