Use the elasticsearch_plugin module for ansible for plugin management

[barryib]

The idea is to use the elasticsearch_plugin module available in ansible 2.0 to manage plugins.

See #124 and #115 for previous discussions

[strootman]

Version of the plugin to be installed. If plugin exists with previous version, it will NOT be updated

So, only the Remove elasticsearch plugins and Install elasticsearch plugins tasks will be replaced?

[barryib]

@strootman yes at the first time. But I have made a feature PR to add force option which remove an install plugin if set to yes. If merged, it should be available in Ansible 2.2

[gingerwizard]

@barryib where are we on using this? Any blockers?

[barryib]

@gingerwizard The PR doesn't merged yet. Still waiting the maintainer @ThePixelDeveloper to review it.

So for now we can probably try on of these solutions:

• Try a workaround with a minimal code change. Like checking if plugin exists and skip the re-installation if es_plugin_reinstall=false
• Include the elasticsearch_plugin as library within the role and use it

I don't really have a strong opinion on these 2 options.

What do you think?

[barryib]

@gingerwizard coming with some updates:

• Fix bug when using proxy #2838 - has been merged and will be available in 2.1.2 and 2.2.0
• Rewrite module to not use unsupported parameters #2839 - has been marked for inclusion

Thanks to @ThePixelDeveloper for its help.

[gingerwizard]

Just waiting for 2.1.2 to be released. Currently rc3. Shield release also pending on this.

[barryib]

@gingerwizard Both ansible's PRs have been merged. I'll make a PR to use the elasticseach_plugin module.

[gingerwizard]

@barryib do you want to take this one on. I think its the next priority now we have 5.x support.

[barryib]

@gingerwizard I can't work on this right now until the March 19th. Actually, I have a limited access to my workspace. But, I've already done all the needed work on the ansible module and fixes and features are shipped with ansible 2.2 and 2.1.2. If it's really urgent, I think someone else should handle it. The only changes are to use the module within the Elasticsearch role. Cheers, Thierno

[gingerwizard]

@barryib i can take this. I dont believe the module handles deltas, however, so we will need to put some ansible logic around the plugin e.g. if the user changes the install list, the role should remove/add the differences. This should be quite simple in 5.x, however, which returns the installed list when plugin list is executed.

[barryib]

@gingerwizard Indeed, the module doesn't support delta. But as you said, playing with ansible filters like: difference and should help. Something like: Plugins to be removed = installed_plugins | difference(es_plugins) Plugins to be installed = es_plugins | difference(removable_plugins) => I don't think that this is necessary if we do plugin install after the plugin removal. Hopefully, the module doesn't install if the plugin is already installed. PS: The next feature I would like to add to the ansible module is the 'force' option which remove and re-install plugin. Cheers, Thierno

[gingerwizard]

I've put together a branch for this i will push once ive added tests. I'm alittle concerned the module doesn't track elasticsearch as tightly as we need. For example, it uses the old proxy flags (for which we already have an issue). By tightly coupling to the module we introduce a dependency which means we can't fix issues until the author of the module addresses them.

As alternative we could possibly bring in the source code to the module - its currently GNU licensed so we'd need to discuss the ramifications for this.

@strootman @jakommo @barryib thoughts please.

[barryib]

@gingerwizard I understand your concern and totally agree with your observation. The main problem, in my opinion, is not really the author but the ansible's release schedule which is almost every 4 months (too long for ansible-elasticsearch).

By matter of principle, I think we should always give back to the community what belongs to the community.

With that said, I think we can bring the source code in as you propose to solve different issues we have. Hopefully, the module is in GPL v3 and its compatible with the Apache license (see here). But we should push back evolutions we'll make to the ansible repo for the community.

Furthermore, the module is pretty simple and quickly removable from the ansible-elacticsearch role when needed (Ex.: when ansible will get a quicker release schedule for they modules 🙈 ).

I'm adding @ThePixelDeveloper, the elasticsearch_module author to have its opinion also.

[barryib]

I have actually a modified version of the elasticsearch_module with a force option which remove and install plugins. Really helpful for plugins update.

[gingerwizard]

unfortunately @barryib this doesn't seem compatible https://www.apache.org/licenses/GPL-compatibility.html

[barryib]

Ouch! right it's only compatible in one way. So that closes the debate. Lets stick to the ES commands for plugins installation.

@gingerwizard I think that we can now close this issue.

[gingerwizard]

Extremely disappointing but it seems we have no choice. Rewriting the module is likely to not produce something enormously different - which makes this more frustrating. Maybe we should move all of the diff logic into a python module as well to produce a distinguishable piece of work. Closing and we'll address this separately.