Configure unicast_hosts dynamically on non-AWS clusters via search

[benattar]

Current cookbook only addresses unicast configuration for systems running under AWS and uses some cleaver logic to populate unicast_hosts using the AWS API. Great solution but it unfortunately leaves those on non-AWS environments without multicast on their networks out in the cold. I have a commit that allows setting the unicast_hosts via an attribute but that sort of defeats the beauty of chef.

I'm opening this issue and would like some feedback from others to solve this intelligently.

My current kludge to fix this is here:
benattar@fbbb82c

Thoughts?

[karmi]

Current cookbook only addresses unicast configuration for systems running under AWS and uses some cleaver logic to populate unicast_hosts using the AWS API.

Could you be more specific, please? Currently, the cookbooks leaves discovery in default settings (multicast), and just configures the AWS environments. I like the solution you did in the attached commit: if the node.elasticsearch[:discovery][:unicast][:hosts] attribute is not empty, switch to unicast and populate the seed list of hosts.

[benattar]

Sure, I can be more specific. My thought process was such that rather than having to rely on manually entering unicast hosts into an attribute we should be able to detect the unicast host based on systems in the same environment with the same clustername specified. When doing this we should also be sensitive to which network interface is utilized for the unicast discovery (should be the same interface we expect to use for publish_host. See pull request https://github.com/karmi/cookbook-elasticsearch/pull/39) for situations where systems may span multiple private networks. I think the above solution is a good starting point. If you agree, we can go that route initially while we flesh out something more extensible and automated. I would like to also do similar unicast host discovery on openstack and rackspace cloud in the same manner as we are doing on AWS hosts in this cookbook. Does that all make sense?

[karmi]

Yes, that makes perfect sense! Node "auto-discovery" could be implemented via the Chef search integration in attributes/recipes, but we need to prevent unnecessary cluster restarts and other problems. On top of setting hosts, we may also support setting discovery.zen.minimum_master_nodes, gateway.expected_nodes and similar settings dynamically.

Note, thate the changes on the configuration branch should now fully support the case from benattar/cookbook-elasticsearch@fbbb82c (setting hosts statically, by configuration).

[benattar]

Yes, I agree, we need better gateway configurations and some better methods to prevent cluster restarts.

[karmi]

To summarize:

• Manually configuring discovery.zen.ping.unicast.hosts is supported in current version
• You can just disable multicast and set the hosts value
• We should add an attribute/recipe file which would perform a search for other Elasticsearch nodes, and add them to discovery.zen.ping.unicast.hosts automatically
• This feature should be opt-in
• We need to make sure we don't restart nodes without reason -- basically any change in the number of nodes would end up restarting the whole cluster. Since unicast hosts are only "seed" values, it would be sufficient to update only the list for the newly created node, and possibly, for any node where Elasticsearch is not running
• Not sure how to test it, without running against Chef

TODOS

• When on Chef Server, perform a search for other nodes, and their IPs them discovery.zen.ping.unicast.hosts
• Add node IPs to configuration for a node being provisioned for the first time
• Add node IPs to configuration on nodes where Elasticsearch service is not running
• Prevent and verify needless cluster restarts at all costs

[josephholsten]

is anyone else working on this? if not, I've got to make this work locally.

[josephholsten]

Wow. Sorry for the spam everyone.

[josephholsten]

Fixed (with doc) in #116

[cvasii]

I think this cookbook still misses something. The scenario is the following: start a chef node with ES, multicast disable, unicast enable. Being the first node, it will search for other nodes in the chef environment, but won't find any, so the unicast hosts will be an empty array. Then we start a second node, which will find the first node, will consider it as master and will have in the unicast hosts the first node ip address. But if you kill the master node and restart it, how will it find the new elected master(the second node)?