New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Multi Node Support for ElasticSearch #61

Open
KnechtionsCoding opened this Issue Nov 15, 2017 · 4 comments

Comments

Projects
None yet
2 participants
@KnechtionsCoding
Contributor

KnechtionsCoding commented Nov 15, 2017

Do we want to support Multi-node installations for Elastic Search? Because I can make the PR, including the updating the FW role that was just added, but only if you want to have that functionality.

it would involve separating out each of the roles on the server side into Kibana, logstash, elasticsearch. It'd also involve separating out the FW, or at least delegating them to the correct place using when conditions.

@sadsfae

This comment has been minimized.

Show comment
Hide comment
@sadsfae

sadsfae Dec 22, 2017

Owner

Hey @KnechtionsCoding I think this would be a great enhancement. I do feel it should be optional and not the default however, configurable via install/group_vars/all.yml. The main reason is that most usage for this playbook is a very quick, easy-to-consume all in one deployment.

I've started thinking about this too and the best way to approach it without adding too much additional complexity on the user end.

Anything you'd care to mock up and have time for would be welcome, we could also consider shipping a multinode branch? Though this functionality should really be a configurable option and done in a way that yaml config options set simply drive it and keep the level of idempotence we have currently.

Thanks for proposing this and all the work you've done so far in improving the stack as a whole.

Owner

sadsfae commented Dec 22, 2017

Hey @KnechtionsCoding I think this would be a great enhancement. I do feel it should be optional and not the default however, configurable via install/group_vars/all.yml. The main reason is that most usage for this playbook is a very quick, easy-to-consume all in one deployment.

I've started thinking about this too and the best way to approach it without adding too much additional complexity on the user end.

Anything you'd care to mock up and have time for would be welcome, we could also consider shipping a multinode branch? Though this functionality should really be a configurable option and done in a way that yaml config options set simply drive it and keep the level of idempotence we have currently.

Thanks for proposing this and all the work you've done so far in improving the stack as a whole.

@KnechtionsCoding

This comment has been minimized.

Show comment
Hide comment
@KnechtionsCoding

KnechtionsCoding Dec 22, 2017

Contributor

Let submit a PR with a quick mockup of what I had before when I did multinode setups and then we can chat about how to achieve it.

I think it can be achieved without a multi node branch. I think it could be configurable easily with the YAML options.

No problem. Glad to be contributing back when I use something in a professional capacity and can help out.

Contributor

KnechtionsCoding commented Dec 22, 2017

Let submit a PR with a quick mockup of what I had before when I did multinode setups and then we can chat about how to achieve it.

I think it can be achieved without a multi node branch. I think it could be configurable easily with the YAML options.

No problem. Glad to be contributing back when I use something in a professional capacity and can help out.

@KnechtionsCoding

This comment has been minimized.

Show comment
Hide comment
@KnechtionsCoding

KnechtionsCoding Dec 22, 2017

Contributor

Submitted #64 so we can continue the conversation of what the logic looks like. Once that is done the config files are easy.

Contributor

KnechtionsCoding commented Dec 22, 2017

Submitted #64 so we can continue the conversation of what the logic looks like. Once that is done the config files are easy.

@sadsfae

This comment has been minimized.

Show comment
Hide comment
@sadsfae

sadsfae Oct 6, 2018

Owner

It's been a while since there's been an update here. I am still planning on getting this implemented it will just need to come second after #66 (Upgrade to 6.4+).

Some notes on multi-node architecture (borrowed from a friend who works at Elastic):

Ideal design is 3 master-eligible nodes and set minimum_master_nodes to 2
and number of data nodes is arbitrary, especially on larger clusters you would have the 3 master nodes separate but for a smaller clusters it's fine to combine them with data nodes.

Obviously this should be configurable as much as it makes sense but that's a good, solid design default for multi-node. I'd also like to re-use as much of the all-in-one code as possible, this may best be in the form of an all.yml config value set for which other nodes should be added, at this point it will still deploy ELK on one specific node but go back and configure any additional nodes as master/data nodes.

One thing to consider here might be setting your default number of shards too as a new configuration item but schema options should probably be best left to the user so long as the minimum for cluster operation is in place.

Owner

sadsfae commented Oct 6, 2018

It's been a while since there's been an update here. I am still planning on getting this implemented it will just need to come second after #66 (Upgrade to 6.4+).

Some notes on multi-node architecture (borrowed from a friend who works at Elastic):

Ideal design is 3 master-eligible nodes and set minimum_master_nodes to 2
and number of data nodes is arbitrary, especially on larger clusters you would have the 3 master nodes separate but for a smaller clusters it's fine to combine them with data nodes.

Obviously this should be configurable as much as it makes sense but that's a good, solid design default for multi-node. I'd also like to re-use as much of the all-in-one code as possible, this may best be in the form of an all.yml config value set for which other nodes should be added, at this point it will still deploy ELK on one specific node but go back and configure any additional nodes as master/data nodes.

One thing to consider here might be setting your default number of shards too as a new configuration item but schema options should probably be best left to the user so long as the minimum for cluster operation is in place.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment