Plugins defined in databag do not get installed

[Pneumatus]

Initially I had been installing plugins via the elasticsearch::plugin recipe with attributes set at role level but have attempted to transition these to a databag. I've created a databag containing:

{
  "id": "plugins",
  "aws-dev": {
    "plugins": {
      "karmi/elasticsearch-paramedic": {},
      "mobz/elasticsearch-head": {},
      "lukas-vlcek/bigdesk": { "version": "2.0.0" },
      "elasticsearch-cloud-aws": { "version": "1.10.0" }
    }
  }
}

The only plugin that gets installed is the 'cloud-aws' plugin (but with the wrong version - 1.11.0 which it appears to get from the attributes/aws.rb). Running chef-client with logging set to debug shows

DEBUG: Plugins list: ["karmi/elasticsearch-paramedic", "mobz/elasticsearch-head", "lukas-vlcek/bigdesk", "elasticsearch-cloud-aws"]

but they do not appear to be converged into node attributes correctly.

[karmi]

@Pneumatus Hi, verified this issue in Vagrant with a plugins databag. Somehow, somewhere, something evidently wipes out the plugin list loaded from the databag.

Can you use a different mechanism to specify plugins?, eg. role override or node attributes?

I'll keep this issue open to investigate it further.

[Pneumatus]

@karmi I've swapped back to using role overrides for now and look forward to a resolution. Cheers.

[karmi]

@Pneumatus Cool. I'm wondering if exposing the plugins list via data bag is a good idea after all, because of higher chance of problems like this. It makes sense to me to support the elasticsearch/settings databag in addition to usual role/node/environment/etc attribute overrides, but would vote for slashing support for "extra" configuration databags. Anybody else any thoughts? //cc @vhyza

[Pneumatus]

@karmi For us at least, having the plugins configured in a databag means we can do all of our environment specific configuration via databags rather than being able to do some in databags and some in environment overrides.

Arguably we could do it all in environment overrides if we wanted (and not use databags at all) if we wanted to keep everything in the same place but I really don't like using environment overrides unless the attributes are for use universally across the environment (like, chef-server URL).

[karmi]

Agreed. Talked about it to @vhyza, and the current situation is a bit too crazy. I think it makes sense to expose:

1. Override attributes via standard Chef means (environment/role/node/etc)
2. Allow using the elasticsearch/settings databag to save all config info pertaining to Elasticsearch -- plugins would obviously fit here nicely.
3. Allow using the elasticsearch/aws and elasticsearch/data databags to save all config info for Amazon and persistence, making it symmetric to elasticsearch::aws and elasticsearch::data recipes
4. Rename the elasticsearch::users databag to elasticsearch::proxy and allow setting any Nginx proxy related attribute there.

[NickPadilla]

@karmi I think i have found a bug in the plugins.rb recipe. It looks like it always just goes to next b/c of the aws check. Attempting to fix now, if I can get it all to work I'll create a pull request. I am using the plugins.json data bag and that is working well, couldn't get the plugins attribute to work in the settings.json - just FYI.

[NickPadilla]

additional FYI.. I am using chef 11.

[NickPadilla]

It seems to me that the aws recipe check shouldn't be there, since we should only install that plugin when the elasticsearch::aws recipe is used. If we find that plugin in the plugins.rb we should just ignore all together and not check for the recipe - since it doesn't really matter. Please let me know if this logic makes sense or if there is a use case for having this logic. Thanks!

[NickPadilla]

Okay, so found the issue here. And i now understand the need for the aws check, even though it could be done differently. Problem is how the specified plugins are added to the default.elasticsearch.plugins field. I'll create a pull request so that we can get this issue resolved.

[NickPadilla]

@karmi Hey Karel - I was hoping we could get this merged soon, or adjusted to work best for you guys. I'd like to not have to rely on my branch, which will not be merged from upstream consistently. Any help would be greatly appreciated, willing to help iron out any problems with the pull request. Thanks for your time/help!

[karmi]

@NickPadilla Thanks for pinging me, left a note at #150 -- just notice above that in an ideal world where we have all the time we need, I'd refactor the support a bit, and move plugins into the elasticsearch/settings databag.

[NickPadilla]

Hey all, this should be fixed. If anyone would like to have the plugins specified in the elasticsearch/settings databag then it would be helpful to add that functionality. Right now it works to use the plugins databag.

[frizop]

Could we have this added to the README.md? Right now I'm unable to get this to work because I don't understand the directory structure. ie:

$ cat data_bags/elasticsearch/plugins.json
{
  "id": "plugins",
  "elasticsearch": {
    "plugins": {
      "karmi/elasticsearch-paramedic": {},
      "mobz/elasticsearch-head": {},
      "lukas-vlcek/bigdesk": { "version": "2.0.0" },
      "elasticsearch-cloud-aws": { "version": "1.10.0" }
    }
  }
}

I'm chef solo with a role like the following:

  "recipe[elasticsearch]",
  "recipe[elasticsearch::plugins]"

And I'm able to make this work using a wrapper recipe and the following overrides, but I'd prefer if I could just throw a databag item in there and not have the extra overhead of another recipe:

override[:elasticsearch][:plugins] = {"mobz/elasticsearch-head" => ""}

etc etc.

[martinb3]

Hello! It sounds like you were able to get this working. In the meantime, we've re-written the cookbook using libraries that expose resources and providers, so you shouldn't see this problem on the newer version. Hope this helps!