systemd-pre-exec script fails silently

[royaldark]

On systemd-based distros (CentOS in this case), there is a script bin/elasticsearch-systemd-pre-exec which runs before starting the main process. This script ensures the CONF_FILE setting is empty, as it has apparently been removed. However, the error message is echoed to stdout, not stderr, so systemd does not pick up the message, resulting in a very confusing, silent failure.

$ systemctl start elasticsearch
Job for elasticsearch.service failed because the control process exited with error code. See "systemctl status elasticsearch.service" and "journalctl -xe" for details.

$ systemctl status elasticsearch
● elasticsearch.service - Elasticsearch
   Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; disabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Wed 2016-01-20 14:42:39 CST; 3s ago
     Docs: http://www.elastic.co
  Process: 10768 ExecStartPre=/usr/share/elasticsearch/bin/elasticsearch-systemd-pre-exec (code=exited, status=1/FAILURE)

Jan 20 14:42:39 db_host systemd[1]: Starting Elasticsearch...
Jan 20 14:42:39 db_host systemd[1]: elasticsearch.service: control process exited, code=exited status=1
Jan 20 14:42:39 db_host systemd[1]: Failed to start Elasticsearch.
Jan 20 14:42:39 db_host systemd[1]: Unit elasticsearch.service entered failed state.
Jan 20 14:42:39 db_host systemd[1]: elasticsearch.service failed.

[jasontedor]

In /usr/lib/systemd/system/elasticsearch.service is StandardOutput=null. If you change this to StandardOutput=journal, attempt to start Elasticsearch to reproduce the error, and then use journalctl to check the journal, you will see:

Jan 21 11:55:14 vagrant-ubuntu-vivid-64 elasticsearch-systemd-pre-exec[11751]: CONF_FILE setting is no longer supported. elasticsearch.yml must be placed in the config directory and cannot be renamed.

[royaldark]

Sure, but why is StandardOutput=journal not the default then? It seems this message should be visible with a default installation, especially when an upgrade from < 2.0 is pretty likely to run into this issue.

[jasontedor]

Sure, but why is StandardOutput=journal not the default then?

@royaldark I agree it should be as I think this situation trips a lot of people up. Changing it will produce more logging in the journal than some system operators are use to, but that's an easier problem to diagnose and address than the opposite problem of not having any logging at all at the critical moment when the system is starting up.

I opened #16159.