Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

systemd-pre-exec script fails silently #16134

Closed
royaldark opened this issue Jan 20, 2016 · 3 comments

Comments

Projects
None yet
2 participants
@royaldark
Copy link

commented Jan 20, 2016

On systemd-based distros (CentOS in this case), there is a script bin/elasticsearch-systemd-pre-exec which runs before starting the main process. This script ensures the CONF_FILE setting is empty, as it has apparently been removed. However, the error message is echoed to stdout, not stderr, so systemd does not pick up the message, resulting in a very confusing, silent failure.

$ systemctl start elasticsearch
Job for elasticsearch.service failed because the control process exited with error code. See "systemctl status elasticsearch.service" and "journalctl -xe" for details.

$ systemctl status elasticsearch
● elasticsearch.service - Elasticsearch
   Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; disabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Wed 2016-01-20 14:42:39 CST; 3s ago
     Docs: http://www.elastic.co
  Process: 10768 ExecStartPre=/usr/share/elasticsearch/bin/elasticsearch-systemd-pre-exec (code=exited, status=1/FAILURE)

Jan 20 14:42:39 db_host systemd[1]: Starting Elasticsearch...
Jan 20 14:42:39 db_host systemd[1]: elasticsearch.service: control process exited, code=exited status=1
Jan 20 14:42:39 db_host systemd[1]: Failed to start Elasticsearch.
Jan 20 14:42:39 db_host systemd[1]: Unit elasticsearch.service entered failed state.
Jan 20 14:42:39 db_host systemd[1]: elasticsearch.service failed.
@jasontedor

This comment has been minimized.

Copy link
Member

commented Jan 21, 2016

In /usr/lib/systemd/system/elasticsearch.service is StandardOutput=null. If you change this to StandardOutput=journal, attempt to start Elasticsearch to reproduce the error, and then use journalctl to check the journal, you will see:

Jan 21 11:55:14 vagrant-ubuntu-vivid-64 elasticsearch-systemd-pre-exec[11751]: CONF_FILE setting is no longer supported. elasticsearch.yml must be placed in the config directory and cannot be renamed.

@jasontedor jasontedor closed this Jan 21, 2016

@royaldark

This comment has been minimized.

Copy link
Author

commented Jan 21, 2016

Sure, but why is StandardOutput=journal not the default then? It seems this message should be visible with a default installation, especially when an upgrade from < 2.0 is pretty likely to run into this issue.

@jasontedor

This comment has been minimized.

Copy link
Member

commented Jan 21, 2016

Sure, but why is StandardOutput=journal not the default then?

@royaldark I agree it should be as I think this situation trips a lot of people up. Changing it will produce more logging in the journal than some system operators are use to, but that's an easier problem to diagnose and address than the opposite problem of not having any logging at all at the critical moment when the system is starting up.

I opened #16159.

jasontedor added a commit that referenced this issue Jan 22, 2016

Merge pull request #16159 from jasontedor/systemd-standard-output
Default standard output to the journal in systemd

Relates #15315, relates #16134
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.