New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

elasticsearch:5.0.0 max virtual memory areas vm.max_map_count [65530] likely too low, increase to at least [262144] #111

Closed
czerasz opened this Issue Jul 15, 2016 · 17 comments

Comments

Projects
None yet
@czerasz
Copy link

czerasz commented Jul 15, 2016

I get this message when I try to run the elasticsearch:5.0.0 image:

$ docker run -it --rm --name=es-test  elasticsearch:5.0.0 
[2016-07-15 09:07:35,914][WARN ][bootstrap                ] unable to install syscall filter: 
java.lang.UnsupportedOperationException: seccomp unavailable: your kernel is buggy and you should upgrade
    at org.elasticsearch.bootstrap.Seccomp.linuxImpl(Seccomp.java:279)
    at org.elasticsearch.bootstrap.Seccomp.init(Seccomp.java:616)
    at org.elasticsearch.bootstrap.JNANatives.trySeccomp(JNANatives.java:215)
    at org.elasticsearch.bootstrap.Natives.trySeccomp(Natives.java:99)
    at org.elasticsearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:94)
    at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:147)
    at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:250)
    at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:96)
    at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:91)
    at org.elasticsearch.cli.SettingCommand.execute(SettingCommand.java:54)
    at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:91)
    at org.elasticsearch.cli.Command.main(Command.java:53)
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:70)
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:63)
[2016-07-15 09:07:36,040][INFO ][node                     ] [Count Abyss] version[5.0.0-alpha4], pid[1], build[3f5b994/2016-06-27T16:23:46.861Z], OS[Linux/4.4.0-28-generic/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_91/25.91-b14]
[2016-07-15 09:07:36,040][INFO ][node                     ] [Count Abyss] initializing ...
[2016-07-15 09:07:36,773][INFO ][plugins                  ] [Count Abyss] modules [percolator, lang-mustache, lang-painless, reindex, aggs-matrix-stats, lang-expression, ingest-common, lang-groovy], plugins []
[2016-07-15 09:07:37,296][INFO ][env                      ] [Count Abyss] using [1] data paths, mounts [[/usr/share/elasticsearch/data (/dev/mapper/ubuntu--vg-root)]], net usable_space [344.8gb], net total_space [446gb], spins? [possibly], types [ext4]
[2016-07-15 09:07:37,297][INFO ][env                      ] [Count Abyss] heap size [1.9gb], compressed ordinary object pointers [true]
[2016-07-15 09:07:38,437][INFO ][node                     ] [Count Abyss] initialized
[2016-07-15 09:07:38,437][INFO ][node                     ] [Count Abyss] starting ...
[2016-07-15 09:07:38,521][INFO ][transport                ] [Count Abyss] publish_address {172.17.0.2:9300}, bound_addresses {[::]:9300}
Exception in thread "main" java.lang.RuntimeException: bootstrap checks failed
initial heap size [268435456] not equal to maximum heap size [2147483648]; this can cause resize pauses and prevents mlockall from locking the entire heap
max virtual memory areas vm.max_map_count [65530] likely too low, increase to at least [262144]
    at org.elasticsearch.bootstrap.BootstrapCheck.check(BootstrapCheck.java:125)
    at org.elasticsearch.bootstrap.BootstrapCheck.check(BootstrapCheck.java:85)
    at org.elasticsearch.bootstrap.BootstrapCheck.check(BootstrapCheck.java:65)
    at org.elasticsearch.bootstrap.Bootstrap$5.validateNodeBeforeAcceptingRequests(Bootstrap.java:178)
    at org.elasticsearch.node.Node.start(Node.java:373)
    at org.elasticsearch.bootstrap.Bootstrap.start(Bootstrap.java:193)
    at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:252)
    at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:96)
    at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:91)
    at org.elasticsearch.cli.SettingCommand.execute(SettingCommand.java:54)
    at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:91)
    at org.elasticsearch.cli.Command.main(Command.java:53)
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:70)
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:63)
Refer to the log for complete error details.
[2016-07-15 09:07:38,528][INFO ][node                     ] [Count Abyss] stopping ...
[2016-07-15 09:07:38,537][INFO ][node                     ] [Count Abyss] stopped
[2016-07-15 09:07:38,538][INFO ][node                     ] [Count Abyss] closing ...
[2016-07-15 09:07:38,546][INFO ][node                     ] [Count Abyss] closed

I also tried this:

$ docker run -it --rm --name=es-test -v $PWD/configuration/elasticsearch/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml elasticsearch:5.0.0
[2016-07-15 09:12:37,798][WARN ][bootstrap                ] unable to install syscall filter: 
java.lang.UnsupportedOperationException: seccomp unavailable: your kernel is buggy and you should upgrade
    at org.elasticsearch.bootstrap.Seccomp.linuxImpl(Seccomp.java:279)
    at org.elasticsearch.bootstrap.Seccomp.init(Seccomp.java:616)
    at org.elasticsearch.bootstrap.JNANatives.trySeccomp(JNANatives.java:215)
    at org.elasticsearch.bootstrap.Natives.trySeccomp(Natives.java:99)
    at org.elasticsearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:94)
    at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:147)
    at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:250)
    at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:96)
    at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:91)
    at org.elasticsearch.cli.SettingCommand.execute(SettingCommand.java:54)
    at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:91)
    at org.elasticsearch.cli.Command.main(Command.java:53)
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:70)
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:63)
[2016-07-15 09:12:37,907][INFO ][node                     ] [winner] version[5.0.0-alpha4], pid[1], build[3f5b994/2016-06-27T16:23:46.861Z], OS[Linux/4.4.0-28-generic/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_91/25.91-b14]
[2016-07-15 09:12:37,907][INFO ][node                     ] [winner] initializing ...
[2016-07-15 09:12:38,754][INFO ][plugins                  ] [winner] modules [percolator, lang-mustache, lang-painless, reindex, aggs-matrix-stats, lang-expression, ingest-common, lang-groovy], plugins []
[2016-07-15 09:12:39,334][INFO ][env                      ] [winner] using [1] data paths, mounts [[/usr/share/elasticsearch/data (/dev/mapper/ubuntu--vg-root)]], net usable_space [344.8gb], net total_space [446gb], spins? [possibly], types [ext4]
[2016-07-15 09:12:39,334][INFO ][env                      ] [winner] heap size [1.9gb], compressed ordinary object pointers [true]
[2016-07-15 09:12:40,509][INFO ][node                     ] [winner] initialized
[2016-07-15 09:12:40,509][INFO ][node                     ] [winner] starting ...
[2016-07-15 09:12:40,592][INFO ][transport                ] [winner] publish_address {172.17.0.2:9300}, bound_addresses {[::]:9300}
Exception in thread "main" java.lang.RuntimeException: bootstrap checks failed
initial heap size [268435456] not equal to maximum heap size [2147483648]; this can cause resize pauses and prevents mlockall from locking the entire heap
max virtual memory areas vm.max_map_count [65530] likely too low, increase to at least [262144]
    at org.elasticsearch.bootstrap.BootstrapCheck.check(BootstrapCheck.java:125)
    at org.elasticsearch.bootstrap.BootstrapCheck.check(BootstrapCheck.java:85)
    at org.elasticsearch.bootstrap.BootstrapCheck.check(BootstrapCheck.java:65)
    at org.elasticsearch.bootstrap.Bootstrap$5.validateNodeBeforeAcceptingRequests(Bootstrap.java:178)
    at org.elasticsearch.node.Node.start(Node.java:373)
    at org.elasticsearch.bootstrap.Bootstrap.start(Bootstrap.java:193)
    at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:252)
    at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:96)
    at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:91)
    at org.elasticsearch.cli.SettingCommand.execute(SettingCommand.java:54)
    at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:91)
    at org.elasticsearch.cli.Command.main(Command.java:53)
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:70)
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:63)
Refer to the log for complete error details.
[2016-07-15 09:12:40,598][INFO ][node                     ] [winner] stopping ...
[2016-07-15 09:12:40,608][INFO ][node                     ] [winner] stopped
[2016-07-15 09:12:40,608][INFO ][node                     ] [winner] closing ...
[2016-07-15 09:12:40,616][INFO ][node                     ] [winner] closed

With the following config:

cluster.name: development
node.name: winner
bootstrap.memory_lock: false
network.host: 0.0.0.0
discovery.zen.minimum_master_nodes: 1
@pwaller

This comment has been minimized.

Copy link

pwaller commented Jul 22, 2016

Same here.

@yosifkit

This comment has been minimized.

Copy link
Member

yosifkit commented Jul 22, 2016

Duplicate of #98,

@yosifkit yosifkit closed this Jul 22, 2016

@fahimbabarpatel

This comment has been minimized.

Copy link

fahimbabarpatel commented Dec 21, 2016

set max_map_count value (Linux)

sudo sysctl -w vm.max_map_count=262144

sagarkar10 added a commit to PriceBoardIn/aws-elk-billing that referenced this issue Dec 23, 2016

@Mohammadtrabelsi

This comment has been minimized.

Copy link

Mohammadtrabelsi commented Dec 23, 2016

From this link The vm_max_map_count kernel setting needs to be set to at least 262144 for production use. Depending on your platform:
Linux

The vm_map_max_count setting should be set permanently in /etc/sysctl.conf:

$ grep vm.max_map_count /etc/sysctl.conf
vm.max_map_count=262144

To apply the setting on a live system type:

sysctl -w vm.max_map_count=262144

ndegory added a commit to appcelerator-archive/amp that referenced this issue Jan 2, 2017

ndegory added a commit to appcelerator-archive/amp that referenced this issue Jan 2, 2017

ndegory added a commit to appcelerator-archive/amp that referenced this issue Jan 2, 2017

ndegory added a commit to appcelerator-archive/amp that referenced this issue Jan 2, 2017

ndegory added a commit to appcelerator-archive/amp that referenced this issue Jan 2, 2017

ndegory added a commit to appcelerator-archive/amp that referenced this issue Jan 2, 2017

ndegory added a commit to appcelerator-archive/amp that referenced this issue Jan 2, 2017

ndegory added a commit to appcelerator-archive/amp that referenced this issue Jan 2, 2017

@DavidBates

This comment has been minimized.

Copy link

DavidBates commented Jan 26, 2017

If you are using this docker image with the official mac version of docker simply go to the docker icon > preferences > advanced and slide the memory to be above 2GB.
image

@tzookb

This comment has been minimized.

Copy link

tzookb commented May 9, 2017

Hey Guys,
I'm on mac and I have increased memory to 6GB

and I still get the same error....

any ideas??

@yosifkit

This comment has been minimized.

Copy link
Member

yosifkit commented May 9, 2017

@tzookb, did you also set max map count like this comment? And possibly an env var of ES_JAVA_OPTS="-Xms1g -Xmx1g"?

@tzookb

This comment has been minimized.

Copy link

tzookb commented May 10, 2017

Thanks @yosifkit

I can add this, to my docker .env file right :

ES_JAVA_OPTS="-Xms1g -Xmx1g"?

but where do I update the above max map?

vm.max_map_count=262144

Thanks!!

@knarfeh

This comment has been minimized.

@Youngyi

This comment has been minimized.

Copy link

Youngyi commented Apr 18, 2018

Linux 16 Azure
please don't forgive open the port 9200 in the Azure.
and copy this command:

sudo sysctl -w vm.max_map_count=262144

@abhilashfancy

This comment has been minimized.

Copy link

abhilashfancy commented Jun 24, 2018

@tzookb you should use the sysctl -w command on the host machine not in the docker container. This will solve the issue. Hope this helps...

@kanihal

This comment has been minimized.

Copy link

kanihal commented Feb 13, 2019

set max_map_count value (Linux)

sudo sysctl -w vm.max_map_count=262144

I don't have sudo access on host machine. So How do I solve this in this case?

@yosifkit

This comment has been minimized.

Copy link
Member

yosifkit commented Feb 13, 2019

@kanihal, If you have full Docker access, you have sudo access. 😉

$ sysctl vm.max_map_count
vm.max_map_count = 262144
$ docker run -it --rm --privileged --net=host --pid=host -v /:/host debian:sid chroot /host sysctl -w vm.max_map_count=262145
vm.max_map_count = 262145
$ sysctl vm.max_map_count
vm.max_map_count = 262145
@kanihal

This comment has been minimized.

Copy link

kanihal commented Feb 15, 2019

@yosifkit Thanks it worked.

host info:

>> lsb_release -a
No LSB modules are available.
Distributor ID:	Debian
Description:	Debian GNU/Linux testing (buster)
Release:	testing
Codename:	buster

sysctl was not in the PATH, it was actually in /sbin/sysctl

BTW why are we using debian:sid? could we use any other debain release?

@yosifkit

This comment has been minimized.

Copy link
Member

yosifkit commented Feb 15, 2019

BTW why are we using debian:sid? could we use any other debain release?

No particular reason; you could use whatever base image you want as long as it has chroot.

@kanihal

This comment has been minimized.

Copy link

kanihal commented Feb 19, 2019

$ docker run -it --rm --privileged --net=host --pid=host -v /:/host debian:sid chroot /host /sbin/sysctl -w vm.max_map_count=262145

@yosifkit
Does this have any side effects of breaking drivers on the host system?
I heard from other users who also use the shared system that nvidia-drivers for GPU stopped working sometime after this because of some kernel mismatch/panic?
Any reason it would happen? Just wondering if it's even possible.

@yosifkit

This comment has been minimized.

Copy link
Member

yosifkit commented Feb 20, 2019

Does this have any side effects of breaking drivers on the host system?

@kanihal, I have no idea. I only know that elasticsearch requires a large enough vm.max_map_count. This can only be done on the host, thus affecting everything running on the system, and would need to be persisted correctly or will be lost on reboot. If this is incompatible with other stuff running on the same box (no clue on nvidia-drivers requirements or if this could cause problems there), then you'll need to run it on a different host.

As the image is working fine then I suggest heading to the Docker Community Forums, the Docker Community Slack, or Stack Overflow for further help or debugging.


As a side note, if you are using the 5 series, then you will have #153 and not run into this problem; that is not true for the the 6 series and above (like the config from 6.6.0). I did manage to get it elasticsearch:6.6.0 to work on a host with a lower vm.max_map_count (this is not a recommended production setup but could work for some limited evaluation and development):

$ sysctl vm.max_map_count
vm.max_map_count = 262000
$ # ie, less than the required 262144
$ # basically this replaces the default cluster ready config
$ # and sets the config in such a way that the bootstrap checks done by elasticsearch are no longer fatal
$ # probably need a volume, not "-it --rm", etcetera
$ docker run -it --rm  elasticsearch:6.6.0 sh -c 'echo "http.host: 0.0.0.0" > /usr/share/elasticsearch/config/elasticsearch.yml && exec docker-entrypoint.sh'
....
[2019-02-20T00:53:20,533][WARN ][o.e.b.BootstrapChecks    ] [qjR9jM-] max virtual memory areas vm.max_map_count [262000] is too low, increase to at least [262144]
....

It would probably be simpler to create an image:

FROM elasticsearch:6.6.0
# cat local-config.yml
# http.host: 0.0.0.0
COPY local-config.yml /usr/share/elasticsearch/config/elasticsearch.yml
# alternative to COPY + file
RUN echo 'http.host: 0.0.0.0' > /usr/share/elasticsearch/config/elasticsearch.yml
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment