Memory locking is not enabled

[micw]

Hi,
your doc recommend to enable memory locking at bootstrap - this is completely missing on the helm chart.

It's a bit tricky on kubernetes but it's possible:

1. enable the required capabilities

      securityContext:
        capabilities:
          add:
          - IPC_LOCK
          - SYS_RESOURCE
   

2. set ulimit before starting elasticsearch

      command:
        - /bin/sh
        - -cxe
        - |
          ulimit -l unlimited
          /usr/local/bin/docker-entrypoint.sh eswrapper
   

3. enable via environment

      env:
        - name: bootstrap.memory_lock
          value: "true"
   

This colud be configurable via helm variable.

[Crazybus]

Hi @micw!

The recommended memory settings is to have swap disabled completely. By default the kubelet disables swapping and I believe it is actually not even possible to have swap enabled inside of Kubernetes container even if you really wanted to (#53533). Memory locking is an alternative method for environments where disabling swap is not possible.

Could you test the output of swapon --summary inside of an Elasticsearch container in your Kubernetes cluster for me? If you do indeed see that swap is enabled could you let me know which Kubernetes version and provider you are using?

[micw]

Swapon won't work within the container, only on the host. I deploy on-premise and usually follow the k8s recommendation not to have swap. But on some environments with long-running processes, I enable swapping and in this environments I'd like to ensure that ES has it's memory reserved.
A good reading about swap on k8s is that answer on stackoverflow: https://serverfault.com/a/931984/236505

Another thing is memory pressure handling: If ES does not reserve memory other pods without memory constraints may be scheduled on the host. When ES starts using the memory, the other pods are moved away. If ES does it's locking, the memory is reserved from beginning.

By default the kubelet disables swapping and I believe it is actually not even possible to have swap enabled inside of Kubernetes container even if you really wanted to (#53533)

Kubelet just checks if it's enabled. It's an option that can be disabled and swap can be added to the host (kubernetes/kubernetes#53533 (comment)).

[Crazybus]

Interesting. Thanks for the extra information.

Kubelet just checks if it's enabled. It's an option that can be disabled and swap can be added to the host (kubernetes/kubernetes#53533 (comment)).

One of the requirements for swap to be enabled in that comment is "containers which do not specify a memory requirement will then by default be able to use all of the machine memory, including swap" . By default a resource request and limit is being set for memory with this chart which would prevent the container from using swap.

Swapon won't work within the container, only on the host

Could you explain what you mean here? When running this in a pod on GKE I can see that swap is not enabled. When I run it in a docker container on a non-kubernetes host with swap enabled I can see swap as expected.

Running locally on my machine:

$ docker run --rm -ti docker.elastic.co/elasticsearch/elasticsearch:6.5.1 sh -c 'swapon --summary'
Filename				Type		Size	Used	Priority
/swap                                  	file	1048572	0	-1

I'm not against adding support memory locking to this chart. I just want to make sure it is actually possible for swap to be enabled inside the container before adding it in.

[botelastic]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[botelastic]

This issue has been automatically closed because it has not had recent activity since being marked as stale.