Handling of core plugins is a pain when using APT repositories to update

[stof]

Elasticsearch version (bin/elasticsearch --version): 5.6.10

Plugins installed: analysis-icu, ingest-attachment

JVM version (java -version):

openjdk version "1.8.0_171"
OpenJDK Runtime Environment (build 1.8.0_171-8u171-b11-0ubuntu0.16.04.1-b11)
OpenJDK 64-Bit Server VM (build 25.171-b11, mixed mode)

OS version: Linux 4.4.0-128-generic #154-Ubuntu SMP Fri May 25 14:15:18 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

Description of the problem including expected versus actual behavior:

When upgrading ES patch releases with APT, core plugins are a pain, because each version is built only for a single release of ES and so ES will fail to start with the new version. And the only way to fix this today is to remove all plugins and install them from scratch again.

I see 3 possible ways to improve this situation (there might be others):

• provide APT packages for core plugins, so that APT would upgrade the plugins at the same time than upgrading ES (and so versions would match)
• provide a CLI command to easily update all installed plugins (or at least all plugins incompatible with current ES version)
• loosen the version checks for core plugins to allow installing them alongside newer patch releases of the same ES minor version at least (not sure this is actually working regarding compatibility).

Steps to reproduce:

1. Install the APT package
2. Install core plugins (analysis-icu and ingest-attachment) in my case
3. Update to the next patch release using APT

ES will fail to start in my case.

[elasticmachine]

Pinging @elastic/es-core-infra

[rjernst]

Providing plugins via deb/rpm packages has been proposed before, most recently in #28762. There are a number of technical challenges there that we not yet ready to complicate plugins with.

Updating the installed plugins via the cli is not feasible, currently, because we do not maintain any information about where the plugin was installed from. To elasticsearch, plugins are all the same: subdirectories in the plugins directory. This would be a large change (needing to keep additional metadata about the plugin, and would also only work with official plugins). Similar to rpm/deb packages for plugins, we are not yet ready to bear this burden, as the need for it has not come up often enough.

Finally, in regards to loosening the version checks: this has been discussed many times since the strict checking was first added in Elasticsearch 2.0. It is something we very much want to do, but will not do it until there is a stable and separate plugin API. Currently, the plugin api relies heavily on internal classes that may change at any time, in any version (including bug fix versions). Before 2.0, these types of changes could cause subtle bugs or horrible upgrade problems. We are working towards a stable plugin api, but it is a very long process that involves cleanly separating out each plugin api, once at a time.

Given that none of these are immediate priorities, I am going to close this issue. Note that the stable plugin api is something a number of as have been slowly working towards, but given the complexities involved, I don't see this happening for quite a while.

[stof]

In the meantime, I created a script that I can run to easily reinstall the plugins I need (tested only with core plugins): https://gist.github.com/stof/9fc74662a4a50a6c1e67a996c7debb2c