Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

elasticsearch:5.0.0 max virtual memory areas vm.max_map_count [65530] likely too low, increase to at least [262144] #111

Closed
czerasz opened this issue Jul 15, 2016 · 22 comments

Comments

@czerasz
Copy link

czerasz commented Jul 15, 2016

I get this message when I try to run the elasticsearch:5.0.0 image:

$ docker run -it --rm --name=es-test  elasticsearch:5.0.0 
[2016-07-15 09:07:35,914][WARN ][bootstrap                ] unable to install syscall filter: 
java.lang.UnsupportedOperationException: seccomp unavailable: your kernel is buggy and you should upgrade
    at org.elasticsearch.bootstrap.Seccomp.linuxImpl(Seccomp.java:279)
    at org.elasticsearch.bootstrap.Seccomp.init(Seccomp.java:616)
    at org.elasticsearch.bootstrap.JNANatives.trySeccomp(JNANatives.java:215)
    at org.elasticsearch.bootstrap.Natives.trySeccomp(Natives.java:99)
    at org.elasticsearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:94)
    at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:147)
    at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:250)
    at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:96)
    at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:91)
    at org.elasticsearch.cli.SettingCommand.execute(SettingCommand.java:54)
    at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:91)
    at org.elasticsearch.cli.Command.main(Command.java:53)
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:70)
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:63)
[2016-07-15 09:07:36,040][INFO ][node                     ] [Count Abyss] version[5.0.0-alpha4], pid[1], build[3f5b994/2016-06-27T16:23:46.861Z], OS[Linux/4.4.0-28-generic/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_91/25.91-b14]
[2016-07-15 09:07:36,040][INFO ][node                     ] [Count Abyss] initializing ...
[2016-07-15 09:07:36,773][INFO ][plugins                  ] [Count Abyss] modules [percolator, lang-mustache, lang-painless, reindex, aggs-matrix-stats, lang-expression, ingest-common, lang-groovy], plugins []
[2016-07-15 09:07:37,296][INFO ][env                      ] [Count Abyss] using [1] data paths, mounts [[/usr/share/elasticsearch/data (/dev/mapper/ubuntu--vg-root)]], net usable_space [344.8gb], net total_space [446gb], spins? [possibly], types [ext4]
[2016-07-15 09:07:37,297][INFO ][env                      ] [Count Abyss] heap size [1.9gb], compressed ordinary object pointers [true]
[2016-07-15 09:07:38,437][INFO ][node                     ] [Count Abyss] initialized
[2016-07-15 09:07:38,437][INFO ][node                     ] [Count Abyss] starting ...
[2016-07-15 09:07:38,521][INFO ][transport                ] [Count Abyss] publish_address {172.17.0.2:9300}, bound_addresses {[::]:9300}
Exception in thread "main" java.lang.RuntimeException: bootstrap checks failed
initial heap size [268435456] not equal to maximum heap size [2147483648]; this can cause resize pauses and prevents mlockall from locking the entire heap
max virtual memory areas vm.max_map_count [65530] likely too low, increase to at least [262144]
    at org.elasticsearch.bootstrap.BootstrapCheck.check(BootstrapCheck.java:125)
    at org.elasticsearch.bootstrap.BootstrapCheck.check(BootstrapCheck.java:85)
    at org.elasticsearch.bootstrap.BootstrapCheck.check(BootstrapCheck.java:65)
    at org.elasticsearch.bootstrap.Bootstrap$5.validateNodeBeforeAcceptingRequests(Bootstrap.java:178)
    at org.elasticsearch.node.Node.start(Node.java:373)
    at org.elasticsearch.bootstrap.Bootstrap.start(Bootstrap.java:193)
    at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:252)
    at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:96)
    at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:91)
    at org.elasticsearch.cli.SettingCommand.execute(SettingCommand.java:54)
    at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:91)
    at org.elasticsearch.cli.Command.main(Command.java:53)
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:70)
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:63)
Refer to the log for complete error details.
[2016-07-15 09:07:38,528][INFO ][node                     ] [Count Abyss] stopping ...
[2016-07-15 09:07:38,537][INFO ][node                     ] [Count Abyss] stopped
[2016-07-15 09:07:38,538][INFO ][node                     ] [Count Abyss] closing ...
[2016-07-15 09:07:38,546][INFO ][node                     ] [Count Abyss] closed

I also tried this:

$ docker run -it --rm --name=es-test -v $PWD/configuration/elasticsearch/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml elasticsearch:5.0.0
[2016-07-15 09:12:37,798][WARN ][bootstrap                ] unable to install syscall filter: 
java.lang.UnsupportedOperationException: seccomp unavailable: your kernel is buggy and you should upgrade
    at org.elasticsearch.bootstrap.Seccomp.linuxImpl(Seccomp.java:279)
    at org.elasticsearch.bootstrap.Seccomp.init(Seccomp.java:616)
    at org.elasticsearch.bootstrap.JNANatives.trySeccomp(JNANatives.java:215)
    at org.elasticsearch.bootstrap.Natives.trySeccomp(Natives.java:99)
    at org.elasticsearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:94)
    at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:147)
    at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:250)
    at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:96)
    at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:91)
    at org.elasticsearch.cli.SettingCommand.execute(SettingCommand.java:54)
    at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:91)
    at org.elasticsearch.cli.Command.main(Command.java:53)
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:70)
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:63)
[2016-07-15 09:12:37,907][INFO ][node                     ] [winner] version[5.0.0-alpha4], pid[1], build[3f5b994/2016-06-27T16:23:46.861Z], OS[Linux/4.4.0-28-generic/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_91/25.91-b14]
[2016-07-15 09:12:37,907][INFO ][node                     ] [winner] initializing ...
[2016-07-15 09:12:38,754][INFO ][plugins                  ] [winner] modules [percolator, lang-mustache, lang-painless, reindex, aggs-matrix-stats, lang-expression, ingest-common, lang-groovy], plugins []
[2016-07-15 09:12:39,334][INFO ][env                      ] [winner] using [1] data paths, mounts [[/usr/share/elasticsearch/data (/dev/mapper/ubuntu--vg-root)]], net usable_space [344.8gb], net total_space [446gb], spins? [possibly], types [ext4]
[2016-07-15 09:12:39,334][INFO ][env                      ] [winner] heap size [1.9gb], compressed ordinary object pointers [true]
[2016-07-15 09:12:40,509][INFO ][node                     ] [winner] initialized
[2016-07-15 09:12:40,509][INFO ][node                     ] [winner] starting ...
[2016-07-15 09:12:40,592][INFO ][transport                ] [winner] publish_address {172.17.0.2:9300}, bound_addresses {[::]:9300}
Exception in thread "main" java.lang.RuntimeException: bootstrap checks failed
initial heap size [268435456] not equal to maximum heap size [2147483648]; this can cause resize pauses and prevents mlockall from locking the entire heap
max virtual memory areas vm.max_map_count [65530] likely too low, increase to at least [262144]
    at org.elasticsearch.bootstrap.BootstrapCheck.check(BootstrapCheck.java:125)
    at org.elasticsearch.bootstrap.BootstrapCheck.check(BootstrapCheck.java:85)
    at org.elasticsearch.bootstrap.BootstrapCheck.check(BootstrapCheck.java:65)
    at org.elasticsearch.bootstrap.Bootstrap$5.validateNodeBeforeAcceptingRequests(Bootstrap.java:178)
    at org.elasticsearch.node.Node.start(Node.java:373)
    at org.elasticsearch.bootstrap.Bootstrap.start(Bootstrap.java:193)
    at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:252)
    at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:96)
    at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:91)
    at org.elasticsearch.cli.SettingCommand.execute(SettingCommand.java:54)
    at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:91)
    at org.elasticsearch.cli.Command.main(Command.java:53)
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:70)
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:63)
Refer to the log for complete error details.
[2016-07-15 09:12:40,598][INFO ][node                     ] [winner] stopping ...
[2016-07-15 09:12:40,608][INFO ][node                     ] [winner] stopped
[2016-07-15 09:12:40,608][INFO ][node                     ] [winner] closing ...
[2016-07-15 09:12:40,616][INFO ][node                     ] [winner] closed

With the following config:

cluster.name: development
node.name: winner
bootstrap.memory_lock: false
network.host: 0.0.0.0
discovery.zen.minimum_master_nodes: 1
@pwaller
Copy link

pwaller commented Jul 22, 2016

Same here.

@yosifkit
Copy link
Member

Duplicate of #98,

@fahimbabarpatel
Copy link

set max_map_count value (Linux)

sudo sysctl -w vm.max_map_count=262144

sagarkar10 added a commit to 300Geeks/aws-elk-billing that referenced this issue Dec 23, 2016
@Mohammadtrabelsi
Copy link

Mohammadtrabelsi commented Dec 23, 2016

From this link The vm_max_map_count kernel setting needs to be set to at least 262144 for production use. Depending on your platform:
Linux

The vm_map_max_count setting should be set permanently in /etc/sysctl.conf:

$ grep vm.max_map_count /etc/sysctl.conf
vm.max_map_count=262144

To apply the setting on a live system type:

sysctl -w vm.max_map_count=262144

ndegory added a commit to appcelerator-archive/amp that referenced this issue Jan 2, 2017
ndegory added a commit to appcelerator-archive/amp that referenced this issue Jan 2, 2017
ndegory added a commit to appcelerator-archive/amp that referenced this issue Jan 2, 2017
ndegory added a commit to appcelerator-archive/amp that referenced this issue Jan 2, 2017
ndegory added a commit to appcelerator-archive/amp that referenced this issue Jan 2, 2017
ndegory added a commit to appcelerator-archive/amp that referenced this issue Jan 2, 2017
ndegory added a commit to appcelerator-archive/amp that referenced this issue Jan 2, 2017
ndegory added a commit to appcelerator-archive/amp that referenced this issue Jan 2, 2017
@DavidBates
Copy link

If you are using this docker image with the official mac version of docker simply go to the docker icon > preferences > advanced and slide the memory to be above 2GB.
image

@tzookb
Copy link

tzookb commented May 9, 2017

Hey Guys,
I'm on mac and I have increased memory to 6GB

and I still get the same error....

any ideas??

@yosifkit
Copy link
Member

yosifkit commented May 9, 2017

@tzookb, did you also set max map count like this comment? And possibly an env var of ES_JAVA_OPTS="-Xms1g -Xmx1g"?

@tzookb
Copy link

tzookb commented May 10, 2017

Thanks @yosifkit

I can add this, to my docker .env file right :

ES_JAVA_OPTS="-Xms1g -Xmx1g"?

but where do I update the above max map?

vm.max_map_count=262144

Thanks!!

@knarfeh
Copy link

knarfeh commented May 10, 2017

@tzookb See here: https://www.elastic.co/guide/en/elasticsearch/reference/current/docker.html#docker-cli-run-prod-mode

@Youngyi
Copy link

Youngyi commented Apr 18, 2018

Linux 16 Azure
please don't forgive open the port 9200 in the Azure.
and copy this command:

sudo sysctl -w vm.max_map_count=262144

@abhilash-behera
Copy link

@tzookb you should use the sysctl -w command on the host machine not in the docker container. This will solve the issue. Hope this helps...

@kanihal
Copy link

kanihal commented Feb 13, 2019

set max_map_count value (Linux)

sudo sysctl -w vm.max_map_count=262144

I don't have sudo access on host machine. So How do I solve this in this case?

@yosifkit
Copy link
Member

@kanihal, If you have full Docker access, you have sudo access. 😉

$ sysctl vm.max_map_count
vm.max_map_count = 262144
$ docker run -it --rm --privileged --net=host --pid=host -v /:/host debian:sid chroot /host sysctl -w vm.max_map_count=262145
vm.max_map_count = 262145
$ sysctl vm.max_map_count
vm.max_map_count = 262145

@kanihal
Copy link

kanihal commented Feb 15, 2019

@yosifkit Thanks it worked.

host info:

>> lsb_release -a
No LSB modules are available.
Distributor ID:	Debian
Description:	Debian GNU/Linux testing (buster)
Release:	testing
Codename:	buster

sysctl was not in the PATH, it was actually in /sbin/sysctl

BTW why are we using debian:sid? could we use any other debain release?

@yosifkit
Copy link
Member

BTW why are we using debian:sid? could we use any other debain release?

No particular reason; you could use whatever base image you want as long as it has chroot.

@kanihal
Copy link

kanihal commented Feb 19, 2019

$ docker run -it --rm --privileged --net=host --pid=host -v /:/host debian:sid chroot /host /sbin/sysctl -w vm.max_map_count=262145

@yosifkit
Does this have any side effects of breaking drivers on the host system?
I heard from other users who also use the shared system that nvidia-drivers for GPU stopped working sometime after this because of some kernel mismatch/panic?
Any reason it would happen? Just wondering if it's even possible.

@yosifkit
Copy link
Member

Does this have any side effects of breaking drivers on the host system?

@kanihal, I have no idea. I only know that elasticsearch requires a large enough vm.max_map_count. This can only be done on the host, thus affecting everything running on the system, and would need to be persisted correctly or will be lost on reboot. If this is incompatible with other stuff running on the same box (no clue on nvidia-drivers requirements or if this could cause problems there), then you'll need to run it on a different host.

As the image is working fine then I suggest heading to the Docker Community Forums, the Docker Community Slack, or Stack Overflow for further help or debugging.


As a side note, if you are using the 5 series, then you will have #153 and not run into this problem; that is not true for the the 6 series and above (like the config from 6.6.0). I did manage to get it elasticsearch:6.6.0 to work on a host with a lower vm.max_map_count (this is not a recommended production setup but could work for some limited evaluation and development):

$ sysctl vm.max_map_count
vm.max_map_count = 262000
$ # ie, less than the required 262144
$ # basically this replaces the default cluster ready config
$ # and sets the config in such a way that the bootstrap checks done by elasticsearch are no longer fatal
$ # probably need a volume, not "-it --rm", etcetera
$ docker run -it --rm  elasticsearch:6.6.0 sh -c 'echo "http.host: 0.0.0.0" > /usr/share/elasticsearch/config/elasticsearch.yml && exec docker-entrypoint.sh'
....
[2019-02-20T00:53:20,533][WARN ][o.e.b.BootstrapChecks    ] [qjR9jM-] max virtual memory areas vm.max_map_count [262000] is too low, increase to at least [262144]
....

It would probably be simpler to create an image:

FROM elasticsearch:6.6.0
# cat local-config.yml
# http.host: 0.0.0.0
COPY local-config.yml /usr/share/elasticsearch/config/elasticsearch.yml
# alternative to COPY + file
RUN echo 'http.host: 0.0.0.0' > /usr/share/elasticsearch/config/elasticsearch.yml

@lkiesow
Copy link

lkiesow commented Mar 7, 2019

If someone else stumbles over this for a test system you can also disable the bootstrap checks by setting in the elasticsearch.yml:

discovery.type: single-node

@Hanhan1989
Copy link

Hanhan1989 commented Apr 7, 2019

If we are using Docker in Windows Home, we have to do the following:

docker-machine ssh
sudo sysctl -w vm.max_map_count=262144

@filipponova
Copy link

filipponova commented May 7, 2019

My docker compose file functional!

services:
  es01:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.0.1
    container_name: es01
    environment:
      - node.name=es01
      - discovery.seed_hosts=es02
      - cluster.initial_master_nodes=es01,es02
      - cluster.name=docker-cluster
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms1g -Xmx1g"
    ulimits:
      nofile:
        soft: 65535
        hard: 65535  
      memlock:
        soft: -1
        hard: -1
    volumes:
      - esdata01:/usr/share/elasticsearch/data
    ports:
      - 9200:9200
    networks:
      - esnet
  es02:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.0.1
    container_name: es02
    environment:
      - node.name=es02
      - discovery.seed_hosts=es01
      - cluster.initial_master_nodes=es01,es02
      - cluster.name=docker-cluster
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms1g -Xmx1g"
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: 65536
        hard: 65536
    volumes:
      - esdata02:/usr/share/elasticsearch/data
    networks:
      - esnet

volumes:
  esdata01:
    driver: local
  esdata02:
    driver: local

networks:
  esnet:```

@ORESoftware
Copy link

how do I run this:

sudo sysctl -w vm.max_map_count=262144

right before my docker containers start up (imagine that I reboot an ec2 instance). does docker-compose have a "before" hook?

@tianon
Copy link
Member

tianon commented Sep 27, 2019

I'd recommend using a file in /etc/sysctl.d to set that value persistently.

In the future, these sorts of questions/requests would be more appropriately posted to the Docker Community Forums, the Docker Community Slack, or Stack Overflow.

@docker-library docker-library locked as resolved and limited conversation to collaborators Sep 27, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests