Not able to set the bootstrap.memory_lock: true in Docker swarm mode

[jkelastic]

Bug Description

docker.elastic.co/elasticsearch/elasticsearch-platinum:6.2.0

  "os" : {
    "refresh_interval" : "1s",
    "refresh_interval_in_millis" : 1000,
    "name" : "Linux",
    "arch" : "amd64",
    "version" : "4.4.0-109-generic",
    "available_processors" : 48,
    "allocated_processors" : 48

Link to Github issues (if available):

Products/Versions (include any installed plugins): ES 6.0

Details - Description: When setting the bootstrap.memory_lock=true in docker swarm mode, it causes ES to crash.

In docker-compose and we get the error below. There is enough ram on the node.

ERROR:

[2018-02-16T18:33:53,937][WARN ][o.e.b.JNANatives         ] Unable to lock JVM Memory: error=12, reason=Cannot allocate memory

[2018-02-16T18:33:53,941][WARN ][o.e.b.JNANatives         ] This can result in part of the JVM being swapped out.

[2018-02-16T18:33:53,941][WARN ][o.e.b.JNANatives         ] Increase RLIMIT_MEMLOCK, soft limit: 65536, hard limit: 65536

[2018-02-16T18:33:53,941][WARN ][o.e.b.JNANatives         ] These can be adjusted by modifying /etc/security/limits.conf, for example: 

# allow user 'elasticsearch' mlockall

elasticsearch soft memlock unlimited

elasticsearch hard memlock unlimited

Docker-compose settings:

elasticsearch-dev-1:

image: docker.elastic.co/elasticsearch/elasticsearch-platinum:6.2.0

environment:

- cluster.name=dce-dev-elasticsearch-cluster

- bootstrap.memory_lock=false # true causes crash

- "ES_JAVA_OPTS=-Xms16g -Xmx16g"

- ELASTIC_PASSWORD=changeme

- xpack.security.transport.ssl.certificate=/usr/share/elasticsearch/config/x-pack/certificates/elasticsearch-dev-1/elasticsearch-dev-1.crt

- xpack.security.transport.ssl.key=/usr/share/elasticsearch/config/x-pack/certificates/elasticsearch-dev-1/elasticsearch-dev-1.key

deploy:

replicas: 1

resources:

limits:

cpus: '8'

memory: 20G # Must be higher than ES_JAVA_OPTS env values

reservations:

cpus: '4'

memory: 2G

placement:

constraints:

# for now deploy to this host only as NFS is available only available on this host

- node.hostname == dcedockdev3

# - node.labels.type == worker

restart_policy:

condition: on-failure

volumes:

- /ssdstor/dce-elk-v6-dev-stack/elasticsearch-dev-1:/usr/share/elasticsearch/data

- /ssdstor/dce-elk-v6-dev-stack/config/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml

- /ssdstor/dce-elk-v6-dev-stack/config/certs:/usr/share/elasticsearch/config/x-pack/certificates

I found the github link below. Could this be a problem?

docker/for-linux#88

Setting ulimit in docker-compose didn't help either.

ulimits:
memlock:
soft: -1
hard: -1

Often, the Docker daemon is started with too low a limit for lockable memory. That's the limit you are seeing in your logs:

Increase RLIMIT_MEMLOCK, soft limit: 65536, hard limit: 65536

The containers inherit this limit, which then becomes a problem for Elasticsearch when it tries lock more memory than allowed.

If you look at your Docker daemon's limits, you will probably see this:

# grep locked /proc/$(ps --no-headers -o pid -C dockerd | tr -d ' ')/limits
Max locked memory         65536                65536                bytes

When we would much prefer to see:

# grep locked /proc/$(ps --no-headers -o pid -C dockerd | tr -d ' ')/limits
Max locked memory         unlimited            unlimited            bytes

It depends on your system how to go about changing the limit, but on my fairly standard Ubuntu system I was able to do this:

echo -e "[Service]\nLimitMEMLOCK=infinity" | SYSTEMD_EDITOR=tee systemctl edit docker.service
systemctl daemon-reload
systemctl restart docker