Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

RPM init.d exits with 0 when JVM fails to start due to not enough memory #14163

Closed
pickypg opened this issue Oct 16, 2015 · 2 comments

Comments

Projects
None yet
4 participants
@pickypg
Copy link
Member

commented Oct 16, 2015

When running the RPM installation of Elasticsearch 1.7.3 (and checking the newer versions of it, likely any release), if you try to give it too much memory, then it claims to succeed to start and it also returns a 0 exit code:

[pickypg@localhost init.d]$ sudo ./elasticsearch start
Starting elasticsearch:                                    [  OK  ]
OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f1abfd30000, 68632248320, 0) failed; error='Cannot allocate memory' (errno=12)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (malloc) failed to allocate 68632248320 bytes for committing reserved memory.
# An error report file with more information is saved as:
# /tmp/jvm-14062/hs_error.log
[pickypg@localhost init.d]$ echo $?
0

As noted by @inqueue, if you pass in -x to the init.d script, then you will find that the daemon starter appears to be to blame, which "succeeds".

@inqueue

This comment has been minimized.

Copy link
Contributor

commented Oct 16, 2015

FWIW, systemd behaves correctly:

# grep HEAP_SIZE= /etc/sysconfig/elasticsearch
# Set ES_HEAP_SIZE to 50% of available RAM, but no more than 31g
ES_HEAP_SIZE=80g

# systemctl start elasticsearch.service
# ^start^status
systemctl status elasticsearch.service
elasticsearch.service - Elasticsearch
   Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled)
   Active: failed (Result: exit-code) since Fri 2015-10-16 11:45:51 EDT; 9s ago
     Docs: http://www.elastic.co
  Process: 11224 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -Des.pidfile=$PID_DIR/elasticsearch.pid -Des.default.path.home=$ES_HOME -Des.default.path.logs=$LOG_DIR -Des.default.path.data=$DATA_DIR -Des.default.config=$CONF_FILE -Des.default.path.conf=$CONF_DIR (code=exited, status=1/FAILURE)
 Main PID: 11224 (code=exited, status=1/FAILURE)

Oct 16 11:45:51 es-mon.zmb.moc systemd[1]: Starting Elasticsearch...
Oct 16 11:45:51 es-mon.zmb.moc systemd[1]: Started Elasticsearch.
Oct 16 11:45:51 es-mon.zmb.moc elasticsearch[11224]: Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_me...=12)
Oct 16 11:45:51 es-mon.zmb.moc systemd[1]: elasticsearch.service: main process exited, code=exited, status=1/FAILURE
Oct 16 11:45:51 es-mon.zmb.moc systemd[1]: Unit elasticsearch.service entered failed state.
Hint: Some lines were ellipsized, use -l to show in full.
@jasontedor

This comment has been minimized.

Copy link
Member

commented Oct 16, 2015

With an RPM built and installed from #14170, we now see the following behavior on a Centos 6.7 system:

$ cat /etc/redhat-release
CentOS release 6.7 (Final)
$ sudo service elasticsearch start
Starting elasticsearch: Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007fda5a660000, 85724889088, 0) failed; error='Cannot allocate memory' (errno=12)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 85724889088 bytes for committing reserved memory.
# An error report file with more information is saved as:
# /tmp/hs_err_pid12673.log
                                                           [FAILED]
$ echo $?
1

instead of the behavior reported in this issue of printing [ OK ] and returning exit status 0.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.