ElasticSearch, NUMA and swap

[clintongormley]

Hiya

There is an interesting article about MySQL and swap when running on linux systems with NUMA architecture - may be relevant to ES as well:

http://jcole.us/blog/archives/2010/09/28/mysql-swap-insanity-and-the-numa-architecture/

The summary is:

with a single large process running on linux with NUMA, it is possible that the process will swap because memory usage is unbalanced between processors, and a single processor's local memory is full.

By starting the command with: numactl --interleave all $command instead, memory usage is balanced and you avoid swapping while memory is available.

[kimchy]

I cam across this article as well, very interesting... . Requires some testing...

[clintongormley]

Handled with bootstrap.mlockall

[cvasii]

If I start elasticsearch with ES_HEAP_SIZE equal to 50% of the RAM on a NUMA architecture with 2 cores is it possible to have issues regarding the 2 nodes that will be created specific to NUMA?

And how exactly boostrap.mlockall handles this? Assuming a box with 16 Gb RAM and two cores, which will lead to Nodes 0 and 1 created by the NUMA system. We don't want swapping and we disable it, but if node 0 exhausted off free memory, node 1 hasn't, and a process scheduled on node 0 needs something, wouldn't this lead to a crash?

We are deploying to a NUMA system (unix OS) with ES_HEAP_SIZE=50% of RAM, bootstrapp.mlockall = true, file descriptors = 64000, ulimit -l unlimited and sometimes the box crashes.

Trying now to get more details from logs, maybe check if the OOM killer logged something and If I found more I will come back with details.