How do I configure multiple nodes on a single server?

[timw077]

elasticsearch_configure 'elasticsearch-data-1' do
  allocated_memory node['CI']['elasticsearch']['Heap_Size']

  configuration ({
    'cluster.name' => node['CI']['elasticsearch']['cluster_name'],
    'transport.tcp.port' => node['CI']['elasticsearch']['Data_1_Transport'],
    'node.name' => "#{node['hostname']}-data-1",
    'http.enabled' => false
  })
  action :manage
end

elasticsearch_configure 'elasticsearch-data-2' do
  allocated_memory node['CI']['elasticsearch']['Heap_Size']
  configuration ({
    'cluster.name' => node['CI']['elasticsearch']['cluster_name'],
    'transport.tcp.port' => node['CI']['elasticsearch']['Data_2_Transport'],
    'node.name' => "#{node['hostname']}-data-2",
    'http.enabled' => false
  })
  action :manage
end

I would have expected /etc/elasticsearch/elasticsearch-data-1.yml and /etc/elasticsearch/elasticsearch-data-2.yml to get created. Instead /etc/elasticsearch/elasticsearch.yml just keeps getting clobbered back and forth.

[jeffbyrnes]

As far as I know, ES maps nodes:servers::1:1, so you would need multiple servers (be they VMs, containers, bare steel). That said, you can run a single-node “cluster” if you like.

[timw077]

@jeffbyrnes I have a cluster that someone built with ansible that has three "nodes" (maybe instances is a better word).

elasticsearch-data-1 0:off 1:off 2:off 3:off 4:off 5:off 6:off
elasticsearch-data-2 0:off 1:off 2:off 3:off 4:off 5:off 6:off
elasticsearch-http 0:off 1:off 2:off 3:off 4:off 5:off 6:off

It has three config files:

elasticsearch-data-1.yml elasticsearch-data-2.yml elasticsearch-http.yml

I would happily copy this config, but I'd like to use chef and this supermarket recipe instead of ansible to build my systems out. (We are using chef for 'everything' else when I inherited this.)

[jeffbyrnes]

Happy to be corrected! I’d suspect your issue lies with the need to give each instance of an elasticsearch_configure a different conf_path, which this resource doesn’t provide.

See lines 136–137 of provider_configure @ v2.2.1 (latest as of this writing) to see that only one elasticsearch.yml is created.

I’d ask though, why multiple nodes on a single machine? Unless you also plan to have each “node” read/write its data to a different disk, I’m unclear what you gain from this approach. We started with a 3-node cluster initially, and have slowly scaled it upwards as disk & memory needs have climbed.

[timw077]

Why have more than one node or instance on a single server? I have more than 30GB of memory, and one instance of ES can only use 30GB of memory.... I didn't spec the hardware, or design the ES configuration. I was told how they should look in the end. We are using ansible today, I'd like to use chef, since that is where we are heading with our Configuration Management.

[jeffbyrnes]

Makes perfect sense, in that case! We designed around horizontal scaling, so my bias runs in the direction of more, smaller, machines.

I don't have any cycles to help change this behavior right now, so I can only encourage you to take a stab at it yourself in a fork, or see if the maintainers can help out.

[karmi]

Hi, just a note that running multiple Elasticsearch instances per node is indeed valid and quite common -- usually in the situation @timw077 describes, where you have a powerful server with GBs of RAM etc. (There are some caveats for this setup, eg. trying to separate master and data nodes, ideally dedicating separate machines (virtual or physical) for masters, so they are not affected by any ripples in the data nodes.)

I bet @martinb3 had this setup in mind when doing the refactoring the cookbook, maybe he can shed more light here -- in any case, it is something the cookbook should support.

[jeffbyrnes]

Love learning new stuff. Thanks for the eye-opening.

[jeffbyrnes]

@timw077 I haven’t sorted it out entirely myself, and have to step away to work on some other stuff, but I think you’ll need to specify a unique instance_name as well as unique paths (see lines 9–40 of resource_configure.rb).

[martinb3]

@timw077 had emailed me about this too. It sounds like we should probably do a blog post and/or README.md entry about it.

[timw077]

Here is my test that continues to fail:
https://github.com/timw077/ES_General

[martinb3]

Hi folks -- just an update: I've been working on this, and found the issue. It has to do with default values for resources being copied around after being modified. I'm working on a fix, and then I'll file an issue describing the problem, as well as a PR to fix it, and a blog post on how to run multiple instances.

[martinb3]

@timw077 can you try the latest release of the cookbook, v2.2.2?

[timw077]

The elasticsearch.yml has the values I expect.

$ grep -R data /etc/elasticsearch-data-/ | sort -k2 -t:
/etc/elasticsearch-data-1/elasticsearch.yml:node.name: ElasticSearch-centos-67-data-1
/etc/elasticsearch-data-2/elasticsearch.yml:node.name: ElasticSearch-centos-67-data-2
/etc/elasticsearch-data-1/elasticsearch.yml:path.conf: /etc/elasticsearch-data-1
/etc/elasticsearch-data-2/elasticsearch.yml:path.conf: /etc/elasticsearch-data-2
/etc/elasticsearch-data-1/elasticsearch.yml:path.data: /data/elasticsearch-data-1
/etc/elasticsearch-data-2/elasticsearch.yml:path.data: /data/elasticsearch-data-2
/etc/elasticsearch-data-1/elasticsearch.yml:path.logs: /var/log/elasticsearch-data-1
/etc/elasticsearch-data-2/elasticsearch.yml:path.logs: /var/log/elasticsearch-data-2

The services start and run.

Thanks!

[martinb3]

Great! Thanks!

[martinb3]

@timw077 I've also placed an example here, if it helps:
https://github.com/elastic/cookbook-elasticsearch/blob/master/test/fixtures/cookbooks/elasticsearch_test/recipes/doubleinstances.rb

[timw077]

I was just fixing my recipe to plagiar... leverage that example.

[martinb3]

I was just fixing my recipe to plagiar... leverage that example.

👍

[timw077]

I don't know if this matters, and since I just learned how to spell elasticsearch a couple weeks ago, I'm not sure I'm in a place to defend one way or another. ;) The guy that set this up, doesn't work here anymore, and we are redoing a lot of his work for a various reasons.

@martinb3 The only thing that seems different to how things are working in production and with the example, is in my kitchen converge, I'm getting:

(cluster_name => kitchen )
/data/elasticsearch/alpha/kitchen/nodes/0
/data/elasticsearch/beta/kitchen/nodes/0

instead of what I have in production:
/data/elasticsearch/data/kitchen/nodes/0
/data/elasticsearch/data/kitchen/nodes/1

I think with a careful mv command, I can fix this while ES is stopped and be okay. All and all, we are all set. Thanks for fixing this.

[martinb3]

@timw077 This should be something you can override by passing a data.dir to each instance's elasticsearch_configure resource. I'm fairly sure this isn't a breaking change, since the use case of two instances literally couldn't have worked successfully before now.

[timw077]

@martinb3

[My production system was setup with ansible. I'm trying to migrate to chef - since we are using it to manage everything else on the systems, and across the board. ]

When I set data.dir, all the instances use:
/data/elasticsearch/data/kitchen/nodes/0 Then the 2nd instance won't start - because of the node.lock.

Both instances are using the same instance id ("0")

The 2nd instance files to start with this error:

[2016-03-25 17:43:35,609][ERROR][bootstrap                ] Exception
org.elasticsearch.ElasticsearchIllegalStateException: Failed to obtain node lock, is the following location writable?: [/usr/share/elasticsearch/kitchen]

The files are writeable...

# find /usr/share/elasticsearch/kitchen  -ls
918881    4 drwxr-xr-x   3 elasticsearch elasticsearch     4096 Mar 25 16:22 /usr/share/elasticsearch/kitchen
918882    4 drwxr-xr-x   3 elasticsearch elasticsearch     4096 Mar 25 16:22 /usr/share/elasticsearch/kitchen/nodes
918883    4 drwxr-xr-x   2 elasticsearch elasticsearch     4096 Mar 25 17:46 /usr/share/elasticsearch/kitchen/nodes/0
918884    0 -rw-r--r--   1 elasticsearch elasticsearch        0 Mar 25 16:22 /usr/share/elasticsearch/kitchen/nodes/0/node.lock

[karmi]

@martinb3, that's great news, thanks!!

[martinb3]

Hi Tim,

In your example before, your grep showed you were seeing the data.dir distinctly for each instance. Your more recent comment/example shows something different, but you took down the example gist you were using.

If you want to put that gist back up, or show me the specific example you're using, I can keep trying to debug it; I'm not sure the provider we wrote has been tested where you're putting the data dir inside the home path (/usr/share/elastic...).

We should probably open a separate issue for this item, though, since the initial reported problem was fixed you had gotten multiple instances working okay.

Cheers,

Martin