so-elasticsearch crashes after upgrade to 2.3.130

[ITfromHell]

Hi guys,

I was running 2.3.120 until upgrade to 2.3.130. Since then, so-elasticsearch crashes every time. The tipps of e.g. #7655 I did already.
Because message via "sudo docker logs so-elasticsearch" at first seconds it runs
Terminating due to java.lang.OutOfMemoryError: Java heap space
I edited value of "esheap" at file "/opt/so/saltstack/local/pillar/minions/_standalone.sls" to "esheap: '5120m'" (because ca. 33% of machines 16GB RAM, see https://docs.securityonion.net/en/2.3/elasticsearch.html?highlight=heap#heap-size). Unfortunately, this setting seems not to help, same error occurs anymore.
Grafana shows some perfomance point (at 300%+) to "so-elasticsearch", until it crashes again.

I also read/tested:
#3932
#4198
#7810
#7655
alot of elastic heap related docu

Environment:
SO 2.3.130 Standalone (updated from .120 via "sudo soup")
HP Workstation Z Series
CPU: Intel(R) Core(TM) i5-4590 CPU @ 3.30GHz, 4 Cores
RAM: 16GB
Filesystem isnt full ;-)

Thanks in advance for your hints.
(Maybe the upgrade isnt the root cause itself just an accidentally correlation)

[dougburks]

How many open indices do you have? You can check as follows:

sudo so-elasticsearch-indices-list

You may need to close or delete some indices to lower Elasticsearch's heap usage:
https://docs.securityonion.net/en/2.3/elasticsearch.html#closing-indices

[ITfromHell]

Hi,
sorry for the late answer, I have been out of office...

sudo so-elasticsearch-indices-list
doesnt show any results, just a new command prompt.

[dougburks]

From #4198 (comment):

The most likely reason for Elasticsearch exiting randomly like that is due to heap or memory issues. The heap is set solely based on your RAM with no knowledge of how much heap is actually necessary for the ingestion rate. So if this is a production deployment monitoring a busy network, it's quite possible that you may need more RAM in the box and/or more heap for Elasticsearch. Please note that 16GB RAM is the bare minimum for a production deployment:
https://docs.securityonion.net/en/2.3/hardware.html#standalone-deployments