New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Service failing to start RHEL 7 #77053
Comments
The error message when running Elasticsearch directly looks like the problem discussed here: https://www.elastic.co/guide/en/elasticsearch/reference/current/executable-jna-tmpdir.html So you can try mounting Regarding the first error, that one certainly is strange. I'd like to resolve the tmp dir issue first, and see if that affects the other problem - they may have the same root cause. |
Pinging @elastic/es-delivery (Team:Delivery) |
@pugnascotia We would prefer to keep /tmp as noexec. Setting the ES_TMPDIR causes elasticsearch to fail when run directly with the same error message as the service above (i.e. NoSuchMethodError) |
For the sake of completeness below is a dump of the output following a direct call after ES_TMPDIR is set:
I have attached the hs_err_pid44781.log file If you require anything else, please let me know. |
You may be running into #77014 - can you also try setting the |
Thanks for the suggestion, but setting the TMPDIR has no effect |
Can you attach the config you have tried, with the environment variables above? |
Contents of /etc/sysconfig/elasticsearch
|
Can I also check that whatever filesystem If so, maybe we need to try using |
/var/lib/elasticsearch/tmp is on a file system with exec, so that doesn't appear to be the issue. As an aside, why would running the process directly without ES_TMPDIR not cause an issue? Dump of strace -f |
Did you set
I suspect this completely disables JNA, rather than enabling it in a semi-broken state. |
Setting TMPDIR appears to have made no difference. See attached the strace dump |
It doesn't look like it's seeing
The check for
We see it hit |
We tried setting the $TMPDIR manually (via export), but this doesn't appear to have made a difference. |
Pinging @elastic/es-core-infra (Team:Core/Infra) |
I don’t think straces are helping here. We know this error is the result of jna trying to load on /tmp with noexec. Can you please provide the Elasticsearch log file? That should help determine what Elasticsearch is seeing. |
Okay elasticsearch.log is empty, but attached are files added to the log folder. |
Are you still starting via bin/elasticsearch directly or using systemd? If the former, what is the exact command line and env you have set? |
Further update. The service actually starts with the TMPDIR set (sorry I didn't try this before). Directly does not. |
Further updates... If we keep TMPDIR, but remove ES_TMPDIR (and LIBFFI_TMPDIR) it works for both direct and service. Therefore, invoking elasticsearch directly with ES_TMPDIR set causes a failure and the service requires the TMPDIR set. |
Today if `libffi` cannot allocate pages of memory which are both writeable and executable then it will attempt to write code to a temporary file. Elasticsearch configures itself a suitable temporary directory for use by JNA but by default `libffi` won't find this directory and will try various other places. In certain configurations, none of the other places that `libffi` tries are suitable. With older versions of JNA this would result in a `SIGSEGV`; since elastic#80617 the JVM will exit with an exception. With this commit we use the `LIBFFI_TMPDIR` environment variable to configure `libffi` to use the same directory as JNA for its temporary files if they are needed. Closes elastic#18272 Closes elastic#73309 Closes elastic#74545 Closes elastic#77014 Closes elastic#77053 Largely supersedes elastic#77285 Relates elastic#80617
Today if `libffi` cannot allocate pages of memory which are both writeable and executable then it will attempt to write code to a temporary file. Elasticsearch configures itself a suitable temporary directory for use by JNA but by default `libffi` won't find this directory and will try various other places. In certain configurations, none of the other places that `libffi` tries are suitable. With older versions of JNA this would result in a `SIGSEGV`; since elastic#80617 the JVM will exit with an exception. With this commit we use the `LIBFFI_TMPDIR` environment variable to configure `libffi` to use the same directory as JNA for its temporary files if they are needed. Closes elastic#18272 Closes elastic#73309 Closes elastic#74545 Closes elastic#77014 Closes elastic#77053 Largely supersedes elastic#77285
Today if `libffi` cannot allocate pages of memory which are both writeable and executable then it will attempt to write code to a temporary file. Elasticsearch configures itself a suitable temporary directory for use by JNA but by default `libffi` won't find this directory and will try various other places. In certain configurations, none of the other places that `libffi` tries are suitable. With older versions of JNA this would result in a `SIGSEGV`; since elastic#80617 the JVM will exit with an exception. With this commit we use the `LIBFFI_TMPDIR` environment variable to configure `libffi` to use the same directory as JNA for its temporary files if they are needed. Closes elastic#18272 Closes elastic#73309 Closes elastic#74545 Closes elastic#77014 Closes elastic#77053 Largely supersedes elastic#77285
Today if `libffi` cannot allocate pages of memory which are both writeable and executable then it will attempt to write code to a temporary file. Elasticsearch configures itself a suitable temporary directory for use by JNA but by default `libffi` won't find this directory and will try various other places. In certain configurations, none of the other places that `libffi` tries are suitable. With older versions of JNA this would result in a `SIGSEGV`; since elastic#80617 the JVM will exit with an exception. With this commit we use the `LIBFFI_TMPDIR` environment variable to configure `libffi` to use the same directory as JNA for its temporary files if they are needed. Closes elastic#18272 Closes elastic#73309 Closes elastic#74545 Closes elastic#77014 Closes elastic#77053 Relates elastic#77285
Today if `libffi` cannot allocate pages of memory which are both writeable and executable then it will attempt to write code to a temporary file. Elasticsearch configures itself a suitable temporary directory for use by JNA but by default `libffi` won't find this directory and will try various other places. In certain configurations, none of the other places that `libffi` tries are suitable. With older versions of JNA this would result in a `SIGSEGV`; since elastic#80617 the JVM will exit with an exception. With this commit we use the `LIBFFI_TMPDIR` environment variable to configure `libffi` to use the same directory as JNA for its temporary files if they are needed. Closes elastic#18272 Closes elastic#73309 Closes elastic#74545 Closes elastic#77014 Closes elastic#77053 Relates elastic#77285
Today if `libffi` cannot allocate pages of memory which are both writeable and executable then it will attempt to write code to a temporary file. Elasticsearch configures itself a suitable temporary directory for use by JNA but by default `libffi` won't find this directory and will try various other places. In certain configurations, none of the other places that `libffi` tries are suitable. With older versions of JNA this would result in a `SIGSEGV`; since elastic#80617 the JVM will exit with an exception. With this commit we use the `LIBFFI_TMPDIR` environment variable to configure `libffi` to use the same directory as JNA for its temporary files if they are needed. Closes elastic#18272 Closes elastic#73309 Closes elastic#74545 Closes elastic#77014 Closes elastic#77053 Relates elastic#77285 Co-authored-by: Rory Hunter <roryhunter2@gmail.com>
Today if `libffi` cannot allocate pages of memory which are both writeable and executable then it will attempt to write code to a temporary file. Elasticsearch configures itself a suitable temporary directory for use by JNA but by default `libffi` won't find this directory and will try various other places. In certain configurations, none of the other places that `libffi` tries are suitable. With older versions of JNA this would result in a `SIGSEGV`; since #80617 the JVM will exit with an exception. With this commit we use the `LIBFFI_TMPDIR` environment variable to configure `libffi` to use the same directory as JNA for its temporary files if they are needed. Closes #18272 Closes #73309 Closes #74545 Closes #77014 Closes #77053 Relates #77285 Co-authored-by: Rory Hunter <roryhunter2@gmail.com>
Today if `libffi` cannot allocate pages of memory which are both writeable and executable then it will attempt to write code to a temporary file. Elasticsearch configures itself a suitable temporary directory for use by JNA but by default `libffi` won't find this directory and will try various other places. In certain configurations, none of the other places that `libffi` tries are suitable. With older versions of JNA this would result in a `SIGSEGV`; since elastic#80617 the JVM will exit with an exception. With this commit we use the `LIBFFI_TMPDIR` environment variable to configure `libffi` to use the same directory as JNA for its temporary files if they are needed. Closes elastic#18272 Closes elastic#73309 Closes elastic#74545 Closes elastic#77014 Closes elastic#77053 Relates elastic#77285 Co-authored-by: Rory Hunter <roryhunter2@gmail.com>
Today if `libffi` cannot allocate pages of memory which are both writeable and executable then it will attempt to write code to a temporary file. Elasticsearch configures itself a suitable temporary directory for use by JNA but by default `libffi` won't find this directory and will try various other places. In certain configurations, none of the other places that `libffi` tries are suitable. With older versions of JNA this would result in a `SIGSEGV`; since elastic#80617 the JVM will exit with an exception. With this commit we use the `LIBFFI_TMPDIR` environment variable to configure `libffi` to use the same directory as JNA for its temporary files if they are needed. Closes elastic#18272 Closes elastic#73309 Closes elastic#74545 Closes elastic#77014 Closes elastic#77053 Relates elastic#77285 Co-authored-by: Rory Hunter <roryhunter2@gmail.com>
Today if `libffi` cannot allocate pages of memory which are both writeable and executable then it will attempt to write code to a temporary file. Elasticsearch configures itself a suitable temporary directory for use by JNA but by default `libffi` won't find this directory and will try various other places. In certain configurations, none of the other places that `libffi` tries are suitable. With older versions of JNA this would result in a `SIGSEGV`; since #80617 the JVM will exit with an exception. With this commit we use the `LIBFFI_TMPDIR` environment variable to configure `libffi` to use the same directory as JNA for its temporary files if they are needed. Closes #18272 Closes #73309 Closes #74545 Closes #77014 Closes #77053 Relates #77285 Co-authored-by: Rory Hunter <roryhunter2@gmail.com> Co-authored-by: Rory Hunter <roryhunter2@gmail.com>
* Set LIBFFI_TMPDIR at startup (#80651) Today if `libffi` cannot allocate pages of memory which are both writeable and executable then it will attempt to write code to a temporary file. Elasticsearch configures itself a suitable temporary directory for use by JNA but by default `libffi` won't find this directory and will try various other places. In certain configurations, none of the other places that `libffi` tries are suitable. With older versions of JNA this would result in a `SIGSEGV`; since #80617 the JVM will exit with an exception. With this commit we use the `LIBFFI_TMPDIR` environment variable to configure `libffi` to use the same directory as JNA for its temporary files if they are needed. Closes #18272 Closes #73309 Closes #74545 Closes #77014 Closes #77053 Relates #77285 Co-authored-by: Rory Hunter <roryhunter2@gmail.com> * Fix incorrect SSL usage Co-authored-by: Rory Hunter <roryhunter2@gmail.com>
Elasticsearch version (
bin/elasticsearch --version
):Version: 7.14.0, Build: default/rpm/dd5a0a2acaa2045ff9624f3729fc8a6f40835aa1/2021-07-29T20:49:32.864135063Z, JVM: 16.0.1
Plugins installed: []
None
JVM version (
java -version
):Bundled version is:
openjdk version "16.0.1" 2021-04-20
OpenJDK Runtime Environment AdoptOpenJDK-16.0.1+9 (build 16.0.1+9)
OpenJDK 64-Bit Server VM AdoptOpenJDK-16.0.1+9 (build 16.0.1+9, mixed mode, sharing)
OS Version is:
openjdk version "16.0.1" 2021-04-20
OpenJDK Runtime Environment AdoptOpenJDK-16.0.1+9 (build 16.0.1+9)
OpenJDK 64-Bit Server VM AdoptOpenJDK-16.0.1+9 (build 16.0.1+9, mixed mode, sharing)
-- note I have not changed settings, so the bundled version should be the one in use
OS version (
uname -a
if on a Unix-like system):Linux 1025093 3.10.0-957.21.2.el7.x86_64 #1 SMP Tue May 28 09:26:43 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Red Hat Enterprise Linux Server 7.9 (Maipo)
Description of the problem including expected versus actual behavior:
After installing elasticsearch via the "Installing from the RPM repository" instructions on https://www.elastic.co/guide/en/elasticsearch/reference/7.14/rpm.html#rpm-repo we are able to start elasticsearch directly /bin/elasticsearch (with warnings). However when attempting to start the service we receive an error.
Warnings received when starting elastic search directly:
See the link for a discussion regarding this:
https://discuss.elastic.co/t/service-will-not-start-rhe7/282591/5
The current thoughts are that the error is caused as elasticsearch cannot assume the SystemdPlugin is running.
https://github.com/elastic/elasticsearch/blob/master/modules/systemd/src/main/java/org/elasticsearch/systemd/Libsystemd.java#L21-L26
Steps to reproduce:
The text was updated successfully, but these errors were encountered: