Elasticsearch doesn't start after X-Pack installation

[vitalykarasik]

Elasticsearch version: 5.3.2

Plugins installed: ingest-geoip
lang-python
repository-s3
x-pack

JVM version (java -version): 1.8.0_131

OS version (uname -a if on a Unix-like system): Linux 3.13.0-36-generic #63-Ubuntu SMP Wed Sep 3 21:30:07 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

Description of the problem including expected versus actual behavior:
Elasticsearch service doesn't start after installing X-Pack.

Steps to reproduce:

1. installed ELK 5.3.2 packages on Ubuntu 14.0.4 via apt-get, services are running
2. Installed X-Pack by
   /usr/share/elasticsearch/bin/elasticsearch-plugin install x-pack
3. service elasticsearch restart
   Service don't start

Provide logs (if relevant):

Java HotSpot(TM) 64-Bit Server VM/1.8.0_131/25.131-b11] [2017-05-04T07:50:19,715][ERROR][o.e.b.Bootstrap ] Exception java.lang.IllegalStateException: Could not load plugin descriptor for existing plugin [x-pack]. Was the plugin built before 2.0? at org.elasticsearch.plugins.PluginsService.getPluginBundles(PluginsService.java:292) ~[elasticsearch-5.3.2.jar:5.3.2] at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:128) ~[elasticsearch-5.3.2.jar:5.3.2] at org.elasticsearch.node.Node.<init>(Node.java:309) ~[elasticsearch-5.3.2.jar:5.3.2] at org.elasticsearch.node.Node.<init>(Node.java:242) ~[elasticsearch-5.3.2.jar:5.3.2] at org.elasticsearch.bootstrap.Bootstrap$6.<init>(Bootstrap.java:242) ~[elasticsearch-5.3.2.jar:5.3.2] at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[elasticsearch-5.3.2.jar:5.3.2] at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:360) [elasticsearch-5.3.2.jar:5.3.2] at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:123) [elasticsearch-5.3.2.jar:5.3.2] at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:114) [elasticsearch-5.3.2.jar:5.3.2] at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:58) [elasticsearch-5.3.2.jar:5.3.2] at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:122) [elasticsearch-5.3.2.jar:5.3.2] at org.elasticsearch.cli.Command.main(Command.java:88) [elasticsearch-5.3.2.jar:5.3.2] at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:91) [elasticsearch-5.3.2.jar:5.3.2] at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:84) [elasticsearch-5.3.2.jar:5.3.2]

[ywelsch]

Could it be related to https://discuss.elastic.co/t/elasticsearch-x-pack-plugin-breaks-elasticsearch/82051/4 ?

Did you install x-pack as root?
Is this the full stack trace?
Can you show us the content of the plugins directory (with permissions)?

[vitalykarasik]

Ywelsch - many thanks for your answer!
Yes, it was permission - I figured it out just before I read your comment :-)
"chown -R elasticsearch /usr/share/elasticsearch/plugins/x-pack/ " fixed the issue.
Yes, I installed as root. I suggest to update documentation. In fact, we cannot install as "elasticsearch" user because it has "/bin/false" as its shell.

[ywelsch]

I followed the installation instructions here and cannot reproduce the permissions issue, so I'm wondering where things could have gone wrong.

[jasontedor]

It is expected that installation be done with root and this is documented. Installing as the elasticsearch user will simply not work, there are files and directories that need to be written to that the elasticsearch user does not have.

I wonder if you have umask in place that is interfering with our expectations (although I thought we addressed this)? Can you check:

$ umask
0002

and share the output here?

Can you also show all the permissions on the tree starting at elasticsearch:

$ tree -L 3 -p

[vitalykarasik]

Yes, I installed as root. I cannot provide original (after install) ownership/permission details because I already fixed them. I guess it's related to umask - it's "0077" for root user on my hardened Ubuntu. I suggest set ownership and permission for all installed files explicitly, w/o relying on system settings. Thank you both, Vitaly

…

On Thu, May 4, 2017 at 3:16 PM, Jason Tedor ***@***.***> wrote: It is expected that installation be done with root and this is documented <https://www.elastic.co/guide/en/x-pack/5.3/installing-xpack.html#xpack-package-installation>. Installing as the elasticsearch user will simply not work, there are files and directories that need to be written to that the elasticsearch user does not have. I wonder if you have umask in place that is interfering with our expectations (although I thought we addressed this)? Can you check: $ umask 0002 and share the output here? Can you also show all the permissions on the tree starting at elasticsearch: $ tree -L 3 -p — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#24480 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AHRn-3h12M9B9SFosPsUIL065KmWutSaks5r2cGrgaJpZM4NQV7Q> .

[jasontedor]

I guess it's related to umask - it's "0077" for root user on my hardened Ubuntu.

Thank you for confirming this, we will look into whether or not this is indeed the cause.

I suggest set ownership and permission for all installed files explicitly, w/o relying on system settings.

Yes, we do this already, but it appears that we might be missing something somewhere.

[jasontedor]

Thank you for confirming this, we will look into whether or not this is indeed the cause.

I can reproduce this by setting the umask to 0077.

[jasontedor]

I found the bug, but it raises some challenges. No promises on a fix in the short term.

[jasontedor]

I found a clean solution and opened #24527.

[vitalykarasik]

Thank you!

…

On Sat, May 6, 2017 at 5:56 AM, Jason Tedor ***@***.***> wrote: I found a clean solution and opened #24527 <#24527>. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#24480 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AHRn-yGOPYneyK4sDMpBP0JvalTTHbP3ks5r2-FLgaJpZM4NQV7Q> .

[jasontedor]

You're welcome.