Log when cannot allocate memory.

[ThomasdOtreppe]

After debugging the issue at https://discuss.elastic.co/t/elasticsearch-1-7-3-crashed-and-doesnt-restart-no-logs/36080 where ElasticSearch wouldn't start and not indicate anything in the logs or console turned out to be due to memory allocation issue.

As of now, when it fails to start due to memory issues, it will just tell that it failed and no reason will be provided in the logs (or on the screen).

In this case, it should indicate the issue in the logs. Or have add a command in the init.d script to debug such issue such as 'debugstart' where it would display startup/error messages in the console.

[jasontedor]

As of now, when it fails to start due to memory issues, it will just tell that it failed and no reason will be provided in the logs (or on the screen).

I'm perplexed here. The JVM will produce error messages and a log file if it dies because it can't allocate memory. For example, if I try to start Elasticsearch requesting 2g of heap on a system with 1g of physical memory I see:

Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x000000008a660000, 1973026816, 0) failed; error='Cannot allocate memory' (errno=12)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 1973026816 bytes for committing reserved memory.
# An error report file with more information is saved as:
# /tmp/hs_err_pid4572.log

and the first few lines of that log file:

#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 1973026816 bytes for committing reserved memory.
# Possible reasons:
#   The system is out of physical RAM or swap space
#   In 32 bit mode, the process size limit was hit
# Possible solutions:
#   Reduce memory load on the system
#   Increase physical memory or swap space
#   Check if swap backing store is full
#   Use 64 bit Java on a 64 bit OS
#   Decrease Java heap size (-Xmx/-Xms)
#   Decrease number of Java threads
#   Decrease Java thread stack sizes (-Xss)
#   Set larger code cache with -XX:ReservedCodeCacheSize=
# This output file may be truncated or incomplete.
#
#  Out of Memory Error (os_linux.cpp:2627), pid=4572, tid=140018280015616

In this case, it should indicate the issue in the logs.

If it has died or never started, it can not.

Or have add a command in the init.d script to debug such issue such as 'debugstart' where it would display startup/error messages in the console.

By default, the information that I showed above should already be showing. You might want to investigate on your system why this is not the case.

[ThomasdOtreppe]

Jason, it worked before and suddenly crashed then never restarted using 'service elasticsearch start' until I decreased the amount of allocated memory significantly.

I don't know how you are starting it, I'm using ElasticSearch 1.7.3 from Elastic repositories and it didn't log anything at all.

I had no idea it is supposed to log/display it since I never saw it. I have no idea what is wrong and what to look for and if you'd like I can give you access to the system via TeamViewer or something similar so you can see for yourself.

[jasontedor]

I don't know how you are starting it, I'm using ElasticSearch 1.7.3 from Elastic repositories and it didn't log anything at all.

For the example above, I'm also using Elasticsearch from the repositories, same version as you, and I'm starting Elasticsearch as a service on CentOS.

[ThomasdOtreppe]

Just tested a brand new install of ElasticSearch (2.1) on Ubuntu 14.04 and then allocated more memory than the system had and when I restarted it (using service elasticsearch restart), it told me it started but didn't start and didn't log anything.

Here are the changes to the configuration:

network.host: 127.0.0.1
discovery.zen.ping.multicast.enabled: false
node.local: true
bootstrap.mlockall: true

Logging was increased to debug and swapiness was disabled (set to 1).

Could you please reopen the ticket?

[rmuir]

its impossible because es is overconfigurable: we can't setup crashes to go to the logs directory, because you can configure logs directory not just via shell script/env, but also inside elasticsearch.yml.

[ThomasdOtreppe]

So, would it be possible to have a 'debugstart' in the init.d script so that we can see what is wrong when it starts?

[rmuir]

that just adds more ways to run elasticsearch and more options. we need less not more.

[ThomasdOtreppe]

rmuir, that sounds contradictory to what Elastic is doing, they are adding features to ElasticSearch.

Right now, debugging it is very complicated if you are not a developer and know Java well enough.

In my case, it took a week to figure out what the issue was after begging (and if you look at the post in the forum, I searched quite a bit before posting).

Unless there is a serious troubleshooting page that covers such issues (this one doesn't), then there is a need for such option.

[jasontedor]

Just tested a brand new install of ElasticSearch (2.1) on Ubuntu 14.04 and then allocated more memory than the system had and when I restarted it (using service elasticsearch restart), it told me it started but didn't start and didn't log anything.

No matter what is going on with the output, JVMs produce an error log file in /tmp by default on Linux, or you can overwrite with -XX:ErrorFile=path.

[ThomasdOtreppe]

@jasontedor I had no idea. That should be documented on the troubleshooting page.

[jasontedor]

Just tested a brand new install of ElasticSearch (2.1) on Ubuntu 14.04 and then allocated more memory than the system had and when I restarted it (using service elasticsearch restart), it told me it started but didn't start and didn't log anything.

The key to understanding the problem here is to understand systemd logging.

I built an Ubuntu VM, installed openjdk-7-jdk, and Elasticsearch 1.7.3 from the Elastic repository. I started Elasticsearch with a heap configuration that would force the JVM to die with a failure to allocate. I reproduced your issue that no log messages are produced on the console (however, as mentioned, there is a JVM error log in /tmp).

Now, let's check the systemd.exec configuration in /usr/lib/systemd/system/elasticsearch.service and focus on these lines:

# Connects standard output to /dev/null
StandardOutput=null

You can change this to journal as in

# Connects standard output to /dev/null
StandardOutput=journal

then reload the daemon

# /bin/systemctl daemon-reload

and start Elasticsearch (again forcing it to die):

# /bin/systemctl start elasticsearch.service

and finally check the journal:

# journalctl

gives

Dec 08 18:06:38 vagrant-ubuntu-vivid-64 elasticsearch[16129]: OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00000006fad30000, 4207738880, 0) failed; error='Cannot allocate memory' (errno=12)
Dec 08 18:06:38 vagrant-ubuntu-vivid-64 elasticsearch[16129]: #
Dec 08 18:06:38 vagrant-ubuntu-vivid-64 elasticsearch[16129]: # There is insufficient memory for the Java Runtime Environment to continue.
Dec 08 18:06:38 vagrant-ubuntu-vivid-64 elasticsearch[16129]: # Native memory allocation (malloc) failed to allocate 4207738880 bytes for committing reserved memory.
Dec 08 18:06:38 vagrant-ubuntu-vivid-64 elasticsearch[16129]: # An error report file with more information is saved as:
Dec 08 18:06:38 vagrant-ubuntu-vivid-64 elasticsearch[16129]: # /tmp/hs_err_pid16129.log

Make sure that you have the correct permissions when you use journalctl.

Check the man pages for systemd.exec, sytstemd,journald, and journalctl for additional configuration options.

And I do not think that this should be the default lest we spam the journal with Elasticsearch logs during normal operation. But feel free to configure as you see fit.

Could you please reopen the ticket?

With the above, I do not think that there is an issue here.

[jasontedor]

Unless there is a serious troubleshooting page that covers such issues (this one doesn't), then there is a need for such option.

You can configure this as you desire with the systemd configuration.

[ThomasdOtreppe]

It would be nice to put all that in the documentation since it doesn't seem that a lot of people know that (from what you can see in the forum post, nobody ever mentioned any of it). I developed in Java a while back and I'm fairly good with Linux but I didn't know the things you mentioned, @jasontedor

[Oliboy50]

same error, it works just fine with default params on ES 1.x and 2.x, but when I try to start ES 5.1.1, it
fails with:

Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x000000008a660000, 1973026816, 0) failed; error='Cannot allocate memory' (errno=12)

I'm not a Java developer at all, so if someone could explain how to decrease the memory required to run ES 5.x. It would be so nice :)

Note: I just want to use ES 5.x in my development environment. Don't need to run a huge cluster :)

[jasontedor]

The default heap size is 2g. If you need to decrease this, edit the jvm.options file. The docs explain this in more detail.

There are two main reasons that you see a difference between 1.x/2.x and 5.x:

• 1.x and 2.x defaulted to a starting heap size of 256m and a max heap size of 1g while 5.x defaults to a starting and max heap size of 2g
• Elasticsearch 5.x touches all pages of the heap at startup forcing those pages to be allocated from the OS

[Oliboy50]

@jasontedor thanks
I reduced to -Xms1g and -Xmx1g (instead of 2g for both) and it works just as before 👍

[surajr]

Even I reduced to -Xms1g and -Xmx1g and it works just as before

[rkm-ravi94]

Seems like you are using virtual machine with less than 2GB of RAM, but elasticsearch is trying to start with 2 GB of RAM (default for Elasticsearch version 5.X, mentioned in jvmOptions)

Either give more memory to your VM, or change Elasticsearch JVM settings in jvmOptions /etc/elasticsearch/jvm.options and lower the values of the following parameters: -Xms512m -Xmx1g
and it will work (if your VM has less than 1gb of ram then provide -Xmx512m only)