Add playbook to force consul leader election

[langston-barrett]

See discussion on #763, related to #566

[stevendborrelli]

Does this require a consul restart or reload to take effect?

We could also populate this file with the inventory data.

[langston-barrett]

As @ChrisAubuchon noted in #763, "consul.json already has retry_after_leave set. That causes Consul to rejoin."

We could definitely repopulate peers.json with inventory data, but since retry_join is set to that same list of IPs (namely, consul_servers_group), it shouldn't make a difference.

[ryane]

@stevendborrelli, @siddharthist is there any more work needed on this? has it been tested? should we add a small doc explaining why you would use this?

[TanyaCouture]

@ryane @siddharthist OS testing in process. Deploying and installing playbook. Are there other tests that should be run?

[ryane]

@TanyaCouture yes, we definitely want to verify if this playbook will fix a consul cluster that is unable to elect a leader (after a crash, for example)

[TanyaCouture]

1. Successful Openstack installation.
2. The consul log showed: failed to sync remote state: No cluster leader
3. Restarted consul: ansible all -m service -a "name=consul state=restarted" --become
4. The updated consul log showed: No cluster leader
5. Ran the new playbook and restarted consul again
6. The updated consul log still showed: No cluster leader

[langston-barrett]

We should take a look at @abn's playbook for doing this.

[stevendborrelli]

We've had enough folks having issues with consul that adding something similar to @abn's playbook makes sense. I don't think the current PR is sufficient.

[langston-barrett]

@stevendborrelli I'll update ASAP.

[kbroughton]

One thing I had to do in #1338 was modify the peers.json content with a default(groups['all']) for when consul_dc_group is not set. You might want to test with consul_dc_group unset to verify.

{% for host in groups[consul_servers_group] | intersect(groups[consul_dc_group] | default(groups['all'])) %}"{{ hostvars[host].private_ipv4 }}:8300"{% if not loop.last %}, 
{% endif %}{% endfor %}

[ryane]

added an updated version of the playbook that worked for me and for @RaunoVV on gitter. probably could use more testing/validation

[ryane]

this has been successfully tested a couple of times and definitely fixes the issue in some scenarios. It is safe to merge now and if we identify other scenarios where it doesn't work, we can create new issues for it.