Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Configure unicast_hosts dynamically on non-AWS clusters via `search` #40

Closed
benattar opened this issue Dec 12, 2012 · 9 comments

Comments

Projects
None yet
4 participants
@benattar
Copy link

commented Dec 12, 2012

Current cookbook only addresses unicast configuration for systems running under AWS and uses some cleaver logic to populate unicast_hosts using the AWS API. Great solution but it unfortunately leaves those on non-AWS environments without multicast on their networks out in the cold. I have a commit that allows setting the unicast_hosts via an attribute but that sort of defeats the beauty of chef.

I'm opening this issue and would like some feedback from others to solve this intelligently.

My current kludge to fix this is here:
benattar@fbbb82c

Thoughts?

@karmi

This comment has been minimized.

Copy link
Member

commented Dec 12, 2012

Current cookbook only addresses unicast configuration for systems running under AWS and uses some cleaver logic to populate unicast_hosts using the AWS API.

Could you be more specific, please? Currently, the cookbooks leaves discovery in default settings (multicast), and just configures the AWS environments. I like the solution you did in the attached commit: if the node.elasticsearch[:discovery][:unicast][:hosts] attribute is not empty, switch to unicast and populate the seed list of hosts.

@benattar

This comment has been minimized.

Copy link
Author

commented Dec 12, 2012

Sure, I can be more specific. My thought process was such that rather than having to rely on manually entering unicast hosts into an attribute we should be able to detect the unicast host based on systems in the same environment with the same clustername specified. When doing this we should also be sensitive to which network interface is utilized for the unicast discovery (should be the same interface we expect to use for publish_host. See pull request https://github.com/karmi/cookbook-elasticsearch/pull/39) for situations where systems may span multiple private networks. I think the above solution is a good starting point. If you agree, we can go that route initially while we flesh out something more extensible and automated. I would like to also do similar unicast host discovery on openstack and rackspace cloud in the same manner as we are doing on AWS hosts in this cookbook. Does that all make sense?

@karmi

This comment has been minimized.

Copy link
Member

commented Jan 15, 2013

Yes, that makes perfect sense! Node "auto-discovery" could be implemented via the Chef search integration in attributes/recipes, but we need to prevent unnecessary cluster restarts and other problems. On top of setting hosts, we may also support setting discovery.zen.minimum_master_nodes, gateway.expected_nodes and similar settings dynamically.

Note, thate the changes on the configuration branch should now fully support the case from benattar/cookbook-elasticsearch@fbbb82c (setting hosts statically, by configuration).

@benattar

This comment has been minimized.

Copy link
Author

commented Jan 15, 2013

Yes, I agree, we need better gateway configurations and some better methods to prevent cluster restarts.

@karmi

This comment has been minimized.

Copy link
Member

commented Feb 26, 2013

To summarize:

  • Manually configuring discovery.zen.ping.unicast.hosts is supported in current version
  • You can just disable multicast and set the hosts value
  • We should add an attribute/recipe file which would perform a search for other Elasticsearch nodes, and add them to discovery.zen.ping.unicast.hosts automatically
  • This feature should be opt-in
  • We need to make sure we don't restart nodes without reason -- basically any change in the number of nodes would end up restarting the whole cluster. Since unicast hosts are only "seed" values, it would be sufficient to update only the list for the newly created node, and possibly, for any node where Elasticsearch is not running
  • Not sure how to test it, without running against Chef

TODOS

  • When on Chef Server, perform a search for other nodes, and their IPs them discovery.zen.ping.unicast.hosts
  • Add node IPs to configuration for a node being provisioned for the first time
  • Add node IPs to configuration on nodes where Elasticsearch service is not running
  • Prevent and verify needless cluster restarts at all costs
@josephholsten

This comment has been minimized.

Copy link

commented Jul 29, 2013

is anyone else working on this? if not, I've got to make this work locally.

josephholsten added a commit to simplymeasured/chef-elasticsearch that referenced this issue Jul 29, 2013

add chef search discovery. Fixes elastic#40
adds a `search_for_nodes()` helper which searches for other servers.
Defaults to role search within the same chef_environment, but allows for
more complex queries. Also defaults to smart interface selection, using
`cloud.local_ipv4` or `ipaddress` attributes as appropriate, but also
allows for an override.

adds a search_discovery recipe which disables multicast discovery and
populates the unicast hosts with the results of `search_for_nodes()`.

josephholsten added a commit to simplymeasured/chef-elasticsearch that referenced this issue Jul 29, 2013

add chef search discovery. Fixes elastic#40
adds a `search_for_nodes()` helper which searches for other servers.
Defaults to role search within the same chef_environment, but allows for
more complex queries. Also defaults to smart interface selection, using
`cloud.local_ipv4` or `ipaddress` attributes as appropriate, but also
allows for an override.

adds a search_discovery recipe which disables multicast discovery and
populates the unicast hosts with the results of `search_for_nodes()`.

josephholsten added a commit to simplymeasured/chef-elasticsearch that referenced this issue Jul 29, 2013

add chef search discovery. Fixes elastic#40
adds a `search_for_nodes()` helper which searches for other servers.
Defaults to role search within the same chef_environment, but allows for
more complex queries. Also defaults to smart interface selection, using
`cloud.local_ipv4` or `ipaddress` attributes as appropriate, but also
allows for an override.

adds a search_discovery recipe which disables multicast discovery and
populates the unicast hosts with the results of `search_for_nodes()`.

josephholsten added a commit to simplymeasured/chef-elasticsearch that referenced this issue Jul 29, 2013

add chef search discovery. Fixes elastic#40
adds a `search_for_nodes()` helper which searches for other servers.
Defaults to role search within the same chef_environment, but allows for
more complex queries. Also defaults to smart interface selection, using
`cloud.local_ipv4` or `ipaddress` attributes as appropriate, but also
allows for an override.

adds a search_discovery recipe which disables multicast discovery and
populates the unicast hosts with the results of `search_for_nodes()`.

josephholsten added a commit to simplymeasured/chef-elasticsearch that referenced this issue Jul 29, 2013

add chef search discovery. Fixes elastic#40
adds a `search_for_nodes()` helper which searches for other servers.
Defaults to role search within the same chef_environment, but allows for
more complex queries. Also defaults to smart interface selection, using
`cloud.local_ipv4` or `ipaddress` attributes as appropriate, but also
allows for an override.

adds a search_discovery recipe which disables multicast discovery and
populates the unicast hosts with the results of `search_for_nodes()`.

josephholsten added a commit to simplymeasured/chef-elasticsearch that referenced this issue Jul 29, 2013

add chef search discovery. Fixes elastic#40
adds a `search_for_nodes()` helper which searches for other servers.
Defaults to role search within the same chef_environment, but allows for
more complex queries. Also defaults to smart interface selection, using
`cloud.local_ipv4` or `ipaddress` attributes as appropriate, but also
allows for an override.

adds a search_discovery recipe which disables multicast discovery and
populates the unicast hosts with the results of `search_for_nodes()`.

josephholsten added a commit to simplymeasured/chef-elasticsearch that referenced this issue Jul 29, 2013

add chef search discovery. Fixes elastic#40
adds a `search_for_nodes()` helper which searches for other servers.
Defaults to role search within the same chef_environment, but allows for
more complex queries. Also defaults to smart interface selection, using
`cloud.local_ipv4` or `ipaddress` attributes as appropriate, but also
allows for an override.

adds a search_discovery recipe which disables multicast discovery and
populates the unicast hosts with the results of `search_for_nodes()`.

josephholsten added a commit to simplymeasured/chef-elasticsearch that referenced this issue Jul 29, 2013

add chef search discovery. Fixes elastic#40
adds a `search_for_nodes()` helper which searches for other servers.
Defaults to role search within the same chef_environment, but allows for
more complex queries. Also defaults to smart interface selection, using
`cloud.local_ipv4` or `ipaddress` attributes as appropriate, but also
allows for an override.

adds a search_discovery recipe which disables multicast discovery and
populates the unicast hosts with the results of `search_for_nodes()`.
@josephholsten

This comment has been minimized.

Copy link

commented Jul 30, 2013

Wow. Sorry for the spam everyone.

@josephholsten

This comment has been minimized.

Copy link

commented Jul 30, 2013

Fixed (with doc) in #116

@cvasii

This comment has been minimized.

Copy link

commented Sep 16, 2014

I think this cookbook still misses something. The scenario is the following: start a chef node with ES, multicast disable, unicast enable. Being the first node, it will search for other nodes in the chef environment, but won't find any, so the unicast hosts will be an empty array. Then we start a second node, which will find the first node, will consider it as master and will have in the unicast hosts the first node ip address. But if you kill the master node and restart it, how will it find the new elected master(the second node)?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.