Issue supplying datadir as a string, not as array.

[wonkyto]

• Module version: 'elastic-elasticsearch', '5.1.0'
• Puppet version: puppet-3.8.6-2.el6sat.noarch
• OS and version: Red Hat Enterprise Linux Server release 6.7 (Santiago)

Bug description

This isn't necessarily a bug with the Elastic Puppet model, however there is an impact of using this module to configure data-nodes when supplying datadir settings.

Since upgrading to version 5.3.0 we've experienced an issue when configuring our elasticsearch.yml using the following format

path.data:
  - /data/elasticsearch/disk01/data-node-02
  - /data/elasticsearch/disk02/data-node-02
  - /data/elasticsearch/disk03/data-node-02
  - /data/elasticsearch/disk04/data-node-02

Unfortunately there appears to be a bug which also adds an unspecified 5th location of /var/lib/elasticsearch to the path.data array.

[2017-04-10T13:09:22,752][INFO ][o.e.e.NodeEnvironment    ] [HOSTNAME-data-node-02] using [5] data paths, mounts [[/data/elasticsearch/disk04 (/dev/vde1), /data/elasticsearch/disk03 (/dev/vdd1), /data/elasticsearch/disk02 (/dev/vdc1), /var (/dev/mapper/vg_volgrp01-lv_var), /data/elasticsearch/disk01 (/dev/vdb1)]], net usable_space [1tb], net total_space [1.1tb], spins? [possibly], types [ext4]

Apparently this relates to https://github.com/elastic/elasticsearch/pull/8381 however we've only experienced this issue since upgrading to Elasticsearch 5.3.0. The only work around from Elastic support was to write the path.data variable as a string (not array). We were encouraged to NOT use the array format going forward.

path.data: "/data/elasticsearch/disk01/data-node-02, /data/elasticsearch/disk02/data-node-02, /data/elasticsearch/disk03/data-node-02, /data/elasticsearch/disk04/data-node-02"

Now while manually changing the format of the elasticsearch.yml does resolve the issue, the problem with using the Elastic Puppet module is that previously we supplied each datadir as an array

 elasticsearch::instance { $name:
    datadir => ['/data/elasticsearch/disk01/data-node-02', '/data/elasticsearch/disk02/data-node-02', '/data/elasticsearch/disk03/data-node-02', '/data/elasticsearch/disk04/data-node-02']
  }

Now we must supply as a string

 elasticsearch::instance { $name:
    datadir => '/data/elasticsearch/disk01/data-node-02, /data/elasticsearch/disk02/data-node-02, /data/elasticsearch/disk03/data-node-02, /data/elasticsearch/disk04/data-node-02'
  }

But this has the unfortunately side effect that the Elastic Puppet module tries to create the directory location: '/data/elasticsearch/disk01/data-node-02, /data/elasticsearch/disk02/data-node-02, /data/elasticsearch/disk03/data-node-02, /data/elasticsearch/disk04/data-node-02', and fails because this isn't a valid directory.

Error: /Stage[main]/Profile::Elasticsearch::Data_node/Profile::Elasticsearch::Data_node::Instance[data-node-02]/Elasticsearch::Instance[data-node-02]/File[/data/elasticsearch/disk01/data-node-02, /data/elasticsearch/disk02/data-node-02, /data/elasticsearch/disk03/data-node-02, /data/elasticsearch/disk04/data-node-02]/ensure: change from absent to directory failed: Cannot create /data/elasticsearch/disk01/data-node-02, /data/elasticsearch/disk02/data-node-02, /data/elasticsearch/disk03/data-node-02, /data/elasticsearch/disk04/data-node-02; parent directory /data/elasticsearch/disk01/data-node-02, /data/elasticsearch/disk02/data-node-02, /data/elasticsearch/disk03/data-node-02, /data/elasticsearch/disk04 does not exist

We are stuck between a rock and hard place. We can't use the elastic puppet module to configure the nodes because of a bug within Elasticsearch, however the advice from Elastic is to not use the format that the Puppet module uses.

[tylerjl]

Wow, yep, that's pretty serious upstream bug. You're right that in its current state, the module isn't going to be able to support that workaround because of the mkdir exec in the instance.pp file. I'm not super familiar with the behavior of the bug, but does the problem persist if the datadir is specified as an array as normal but the DATA_DIR directory that points to the default directory is set to empty? (with something like init_defaults => { 'DATA_DIR' => '' }

[tylerjl]

@wonkyto (and any other observers), I have a changed staged in my fork that places the instance's config hash at the highest precedence level when merging the final instance configuration. This means that instead of defining datadir => 'foo,bar', you should be able to set config => { 'path.data' => 'foo,bar' } on an elasticsearch::instance and it'll be written in the final config. The datadir parameter mostly serves to handle directory creation for the paths you define - if they already exist, you should be able to set the path.data value manually and override it with a value that doesn't get mkdir-ed and causes the error you're seeing.

If someone can confirm this serves as a viable workaround, I can merge and release it.

[wonkyto]

Hi @tylerjl

Thanks for getting following up and sorry for the delay, I was not available yesterday.

I have tested your fork and I confirm that this works.

Thanks very much for such a quick turn around.

[wonkyto]

Hi Peter, Yes that bug report is spot on. Fortunately we picked this up in model, but only because we are using multiple nodes per physical server - with two nodes trying to write to the same (unspecified) location it was easy to discover because the 2nd node just wouldn't start.

On Tue, 11 Apr 2017, at 02:49 PM, Peter Dyson wrote: Seems it might also be related: elastic/elasticsearch#23981[1] — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub[2], or mute the thread[3].

Links: 1. elastic/elasticsearch#23981 2. #802 (comment) 3. https://github.com/notifications/unsubscribe-auth/AFfx4bPSa5ZMumlak6_eESsuxMC4bE9bks5ruwZugaJpZM4M4SpP

[geekpete]

Related: elastic/elasticsearch#23981

[tylerjl]

Released: https://forge.puppet.com/elastic/elasticsearch/changelog#511-april-13-2017