Cannot start ES 7.0.1 on Fedora 30 unless "xpack.ml.enabled: false" is set

[damianj]

Elasticsearch version (bin/elasticsearch --version): 7.0.1

Plugins installed: N/A (default RPM installation on Fedora)

JVM version (java -version): One that is packaged with ElasticSearch (I haven't changed the JAVA_HOME variable in /etc/sysconfig/elasticsearch, system Java is OpenJDK 12.0.1+12)

OS version (uname -a if on a Unix-like system): Linux fedora-server 5.0.11-300.fc30.x86_64 #1 SMP Thu May 2 14:11:38 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Description of the problem including expected versus actual behavior:
I cannot start elastic search unless I set xpack.ml.enabled: false in /etc/elasticsearch/elasticsearch.yml. I've also tried setting a custom temp directory with all permissions and creating an elasticsearch directory in that at the suggestion of someone from the forums. Note that if I set xpack.ml.enabled: false then I don't need to create a custom temp directory and everything works fine with the default configuration options.

Extra context from forum discussion https://discuss.elastic.co/t/trouble-starting-es-7-0-1-on-fedora-30/179693:

Me: Just wondering if there are known issues with running ES 7.0.1 on Fedora 30? I was unable to start it until I set the xpack.ml.enabled: false in /etc/elasticsearch/elasticsearch.yml. Pretty much running the default settings aside from that. I installed it via the RPM package and am starting it with systemd. The journalctl output is below: Similar Log to the one below in this post

Elastic Team Member: Is it possible, that you are not allowed to write into the /tmp filesystem with the user that is starting Elasticsearch?

Me: I actually thought that at first, but I tried specifying a custom tmp directory via ES_TMPDIR=/elasticsearch_tmp and ES_JAVA_OPTS="-Djna.tmpdir=/elasticsearch_tmp" in /etc/sysconfig/elasticsearch and gave the elasticsearch user and group ownership of it and did chmod 777 to the directory just to make sure there was not doubt it didn't have access. With this it still results in: org.elasticsearch.bootstrap.StartupException: ElasticsearchException[Failed to create native process factories for Machine Learning]; nested: FileNotFoundException[/elasticsearch_tmp/controller_log_4828 (No such file or directory)]; I also see a folder jna--1985354563 in that directory I created, so it is able to access it and create files/folders inside it. After that I reverted the changes to the tmp dir and proceeded to set xpack.ml.enabled: false to get it to work for now while I try and get this sorted.

Elastic Team Member: you can try to create /elasticsearch_tmp/elasticsearch and give that directory all the permissions - this is IIRC where ML needs to have write permissions. If that does not work either, please go ahead and open an issue in the Elasticsearch repo

Steps to reproduce:

1. Install ElasticSearch 7 on Fedora 30 using RPM Repository (basically followed these steps: https://www.elastic.co/guide/en/elasticsearch/reference/current/rpm.html)
2. Start ElasticSearch using sudo systemctl start elasticsearch.service
3. Startup fails

Provide logs (if relevant):

May 06 14:45:13 fedora-server elasticsearch[7428]: [2019-05-06T14:45:13,148][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [fedora-server] uncaught exception in thread [main]
May 06 14:45:13 fedora-server elasticsearch[7428]: org.elasticsearch.bootstrap.StartupException: ElasticsearchException[Failed to create native process factories for Machine Learning]; nested: FileNotFoundException[/elasticsearch_tmp/controller_log_7428 (No such file or directory)];
May 06 14:45:13 fedora-server elasticsearch[7428]:         at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-7.0.1.jar:7.0.1]
May 06 14:45:13 fedora-server elasticsearch[7428]:         at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-7.0.1.jar:7.0.1]
May 06 14:45:13 fedora-server elasticsearch[7428]:         at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-7.0.1.jar:7.0.1]
May 06 14:45:13 fedora-server elasticsearch[7428]:         at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-7.0.1.jar:7.0.1]
May 06 14:45:13 fedora-server elasticsearch[7428]:         at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-7.0.1.jar:7.0.1]
May 06 14:45:13 fedora-server elasticsearch[7428]:         at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) ~[elasticsearch-7.0.1.jar:7.0.1]
May 06 14:45:13 fedora-server elasticsearch[7428]:         at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-7.0.1.jar:7.0.1]
May 06 14:45:13 fedora-server elasticsearch[7428]: Caused by: org.elasticsearch.ElasticsearchException: Failed to create native process factories for Machine Learning
May 06 14:45:13 fedora-server elasticsearch[7428]:         at org.elasticsearch.xpack.ml.MachineLearning.createComponents(MachineLearning.java:433) ~[?:?]
May 06 14:45:13 fedora-server elasticsearch[7428]:         at org.elasticsearch.node.Node.lambda$new$9(Node.java:438) ~[elasticsearch-7.0.1.jar:7.0.1]
May 06 14:45:13 fedora-server elasticsearch[7428]:         at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:271) ~[?:?]
May 06 14:45:13 fedora-server elasticsearch[7428]:         at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1654) ~[?:?]
May 06 14:45:13 fedora-server elasticsearch[7428]:         at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) ~[?:?]
May 06 14:45:13 fedora-server elasticsearch[7428]:         at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) ~[?:?]
May 06 14:45:13 fedora-server elasticsearch[7428]:         at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913) ~[?:?]
May 06 14:45:13 fedora-server elasticsearch[7428]:         at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?]
May 06 14:45:13 fedora-server elasticsearch[7428]:         at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:578) ~[?:?]
May 06 14:45:13 fedora-server elasticsearch[7428]:         at org.elasticsearch.node.Node.<init>(Node.java:441) ~[elasticsearch-7.0.1.jar:7.0.1]
May 06 14:45:13 fedora-server elasticsearch[7428]:         at org.elasticsearch.node.Node.<init>(Node.java:251) ~[elasticsearch-7.0.1.jar:7.0.1]
May 06 14:45:13 fedora-server elasticsearch[7428]:         at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:211) ~[elasticsearch-7.0.1.jar:7.0.1]
May 06 14:45:13 fedora-server elasticsearch[7428]:         at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:211) ~[elasticsearch-7.0.1.jar:7.0.1]
May 06 14:45:13 fedora-server elasticsearch[7428]: [2019-05-06T14:45:13,148][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [fedora-server] uncaught exception in thread [main]
May 06 14:45:13 fedora-server elasticsearch[7428]: org.elasticsearch.bootstrap.StartupException: ElasticsearchException[Failed to create native process factories for Machine Learning]; nested: FileNotFoundException[/elasticsearch_tmp/controller_log_7428 (No such file or directory)];
May 06 14:45:13 fedora-server elasticsearch[7428]: [2019-05-06T14:45:13,148][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [fedora-server] uncaught exception in thread [main]
May 06 14:45:13 fedora-server elasticsearch[7428]: org.elasticsearch.bootstrap.StartupException: ElasticsearchException[Failed to create native process factories for Machine Learning]; nested: FileNotFoundException[/elasticsearch_tmp/controller_log_7428 (No such file or directory)];
May 06 14:45:13 fedora-server elasticsearch[7428]:         at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-7.0.1.jar:7.0.1]
May 06 14:45:13 fedora-server elasticsearch[7428]:         at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-7.0.1.jar:7.0.1]
May 06 14:45:13 fedora-server elasticsearch[7428]:         at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-7.0.1.jar:7.0.1]
May 06 14:45:13 fedora-server elasticsearch[7428]:         at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-7.0.1.jar:7.0.1]
May 06 14:45:13 fedora-server elasticsearch[7428]:         at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-7.0.1.jar:7.0.1]
May 06 14:45:13 fedora-server elasticsearch[7428]:         at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) ~[elasticsearch-7.0.1.jar:7.0.1]
May 06 14:45:13 fedora-server elasticsearch[7428]:         at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-7.0.1.jar:7.0.1]
May 06 14:45:13 fedora-server elasticsearch[7428]: Caused by: org.elasticsearch.ElasticsearchException: Failed to create native process factories for Machine Learning
May 06 14:45:13 fedora-server elasticsearch[7428]:         at org.elasticsearch.xpack.ml.MachineLearning.createComponents(MachineLearning.java:433) ~[?:?]
May 06 14:45:13 fedora-server elasticsearch[7428]:         at org.elasticsearch.node.Node.lambda$new$9(Node.java:438) ~[elasticsearch-7.0.1.jar:7.0.1]
May 06 14:45:13 fedora-server elasticsearch[7428]:         at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:271) ~[?:?]
May 06 14:45:13 fedora-server elasticsearch[7428]:         at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1654) ~[?:?]
May 06 14:45:13 fedora-server elasticsearch[7428]:         at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) ~[?:?]
May 06 14:45:13 fedora-server elasticsearch[7428]:         at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) ~[?:?]
May 06 14:45:13 fedora-server elasticsearch[7428]:         at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913) ~[?:?]
May 06 14:45:13 fedora-server elasticsearch[7428]:         at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?]
May 06 14:45:13 fedora-server elasticsearch[7428]:         at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:578) ~[?:?]
May 06 14:45:13 fedora-server elasticsearch[7428]:         at org.elasticsearch.node.Node.<init>(Node.java:441) ~[elasticsearch-7.0.1.jar:7.0.1]
May 06 14:45:13 fedora-server elasticsearch[7428]:         at org.elasticsearch.node.Node.<init>(Node.java:251) ~[elasticsearch-7.0.1.jar:7.0.1]
May 06 14:45:13 fedora-server elasticsearch[7428]:         at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:211) ~[elasticsearch-7.0.1.jar:7.0.1]
May 06 14:45:13 fedora-server elasticsearch[7428]:         at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:211) ~[elasticsearch-7.0.1.jar:7.0.1]
May 06 14:45:13 fedora-server elasticsearch[7428]:         at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:325) ~[elasticsearch-7.0.1.jar:7.0.1]
May 06 14:45:13 fedora-server elasticsearch[7428]:         at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.0.1.jar:7.0.1]
May 06 14:45:13 fedora-server elasticsearch[7428]:         ... 6 more
May 06 14:45:13 fedora-server elasticsearch[7428]: Caused by: java.io.FileNotFoundException: /elasticsearch_tmp/controller_log_7428 (No such file or directory)
May 06 14:45:13 fedora-server elasticsearch[7428]:         at java.io.FileInputStream.open0(Native Method) ~[?:?]
May 06 14:45:13 fedora-server elasticsearch[7428]:         at java.io.FileInputStream.open(FileInputStream.java:213) ~[?:?]
May 06 14:45:13 fedora-server elasticsearch[7428]:         at java.io.FileInputStream.<init>(FileInputStream.java:155) ~[?:?]
May 06 14:45:13 fedora-server elasticsearch[7428]:         at java.io.FileInputStream.<init>(FileInputStream.java:110) ~[?:?]
May 06 14:45:13 fedora-server elasticsearch[7428]:         at org.elasticsearch.xpack.ml.utils.NamedPipeHelper$PrivilegedInputPipeOpener.run(NamedPipeHelper.java:288) ~[?:?]
May 06 14:45:13 fedora-server elasticsearch[7428]:         at org.elasticsearch.xpack.ml.utils.NamedPipeHelper$PrivilegedInputPipeOpener.run(NamedPipeHelper.java:277) ~[?:?]
May 06 14:45:13 fedora-server elasticsearch[7428]:         at java.security.AccessController.doPrivileged(AccessController.java:310) ~[?:?]
May 06 14:45:13 fedora-server elasticsearch[7428]:         at org.elasticsearch.xpack.ml.utils.NamedPipeHelper.openNamedPipeInputStream(NamedPipeHelper.java:130) ~[?:?]
May 06 14:45:13 fedora-server elasticsearch[7428]:         at org.elasticsearch.xpack.ml.utils.NamedPipeHelper.openNamedPipeInputStream(NamedPipeHelper.java:97) ~[?:?]
May 06 14:45:13 fedora-server elasticsearch[7428]:         at org.elasticsearch.xpack.ml.process.ProcessPipes.connectStreams(ProcessPipes.java:131) ~[?:?]
May 06 14:45:13 fedora-server elasticsearch[7428]:         at org.elasticsearch.xpack.ml.process.NativeController.<init>(NativeController.java:61) ~[?:?]
May 06 14:45:13 fedora-server elasticsearch[7428]:         at org.elasticsearch.xpack.ml.process.NativeControllerHolder.getNativeController(NativeControllerHolder.java:40) ~[?:?]
May 06 14:45:13 fedora-server elasticsearch[7428]:         at org.elasticsearch.xpack.ml.MachineLearning.createComponents(MachineLearning.java:418) ~[?:?]
May 06 14:45:13 fedora-server elasticsearch[7428]:         at org.elasticsearch.node.Node.lambda$new$9(Node.java:438) ~[elasticsearch-7.0.1.jar:7.0.1]
May 06 14:45:13 fedora-server elasticsearch[7428]:         at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:271) ~[?:?]
May 06 14:45:13 fedora-server elasticsearch[7428]:         at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1654) ~[?:?]
May 06 14:45:13 fedora-server elasticsearch[7428]:         at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) ~[?:?]
May 06 14:45:13 fedora-server elasticsearch[7428]:         at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) ~[?:?]
May 06 14:45:13 fedora-server elasticsearch[7428]:         at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913) ~[?:?]
May 06 14:45:13 fedora-server elasticsearch[7428]:         at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?]
May 06 14:45:13 fedora-server elasticsearch[7428]:         at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:578) ~[?:?]
May 06 14:45:13 fedora-server elasticsearch[7428]:         at org.elasticsearch.node.Node.<init>(Node.java:441) ~[elasticsearch-7.0.1.jar:7.0.1]
May 06 14:45:13 fedora-server elasticsearch[7428]:         at org.elasticsearch.node.Node.<init>(Node.java:251) ~[elasticsearch-7.0.1.jar:7.0.1]
May 06 14:45:13 fedora-server elasticsearch[7428]:         at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:211) ~[elasticsearch-7.0.1.jar:7.0.1]
May 06 14:45:13 fedora-server elasticsearch[7428]:         at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:211) ~[elasticsearch-7.0.1.jar:7.0.1]
May 06 14:45:13 fedora-server elasticsearch[7428]:         at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:325) ~[elasticsearch-7.0.1.jar:7.0.1]
May 06 14:45:13 fedora-server elasticsearch[7428]:         at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.0.1.jar:7.0.1]
May 06 14:45:13 fedora-server elasticsearch[7428]:         ... 6 more
May 06 14:45:13 fedora-server systemd[1]: elasticsearch.service: Main process exited, code=exited, status=1/FAILURE
May 06 14:45:13 fedora-server systemd[1]: elasticsearch.service: Failed with result 'exit-code'.

[jasontedor]

I suspect this is elastic/ml-cpp#478. Please try installing the package libxcrypt-compat and let us know if the issue still persists.

[damianj]

Yes, indeed, sudo dnf install libxcrypt-compat fixes the issue. The service starts without needing any special temp directory (or other) configuration. Thank you!

[jasontedor]

You're welcome!

[elasticmachine]

Pinging @elastic/ml-core