New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Vanilla Elasticsearch 5.0.0alpha4 unable to start in LXD container #19458

Closed
naisanza opened this Issue Jul 15, 2016 · 8 comments

Comments

Projects
None yet
4 participants
@naisanza

naisanza commented Jul 15, 2016

Elasticsearch version: 5.0.0alpha4

JVM version: Oracle Java JRE Version 8 Update 91

OS version: Ubuntu 16.04 LXD (Ubuntu 16.04 LTS amd64 image fingerprint: f452cda3bccb)

Description of the problem including expected versus actual behavior:

Running 5.0.0alpha4 elasticsearch fails.
Running 2.3.4 elasticsearch succeeds.

Steps to reproduce:

  1. Launch LXD container
  2. Install Java + Elasticsearch 5.0.0alpha4
  3. Run Elasticsearch

Provide logs (if relevant):

5.0.0alpha4 elasticsearch.yml:

root@base-elasticsearch-5-0-0-alpha4:~# grep -v '#' /opt/elasticsearch/elasticsearch-5.0.0-alpha4/config/elasticsearch.yml
cluster.name: beta
network.host: _site_

2.3.4 elasticsearch.yml:

root@base-elasticsearch-2-3-4:~# grep -v '#' /opt/elasticsearch/elasticsearch-2.3.4/config/elasticsearch.yml
cluster.name: waldo
network.host: _site_

5.0.0alpha4 startup log:

elasticsearch@base-elasticsearch-5-0-0-alpha4:~$ elasticsearch 2>>elasticsearch5.0.0.log 1>>elasticsearch5.0.0.log
elasticsearch@base-elasticsearch-5-0-0-alpha4:~$ cat elasticsearch5.0.0.log
[2016-07-15 20:50:14,763][INFO ][node                     ] [Raa of the Caves] version[5.0.0-alpha4], pid[694], build[3f5b994/2016-06-27T16:23:46.861Z], OS[Linux/4.4.0-28-generic/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_91/25.91-b14]
[2016-07-15 20:50:14,763][INFO ][node                     ] [Raa of the Caves] initializing ...
[2016-07-15 20:50:15,833][INFO ][plugins                  ] [Raa of the Caves] modules [percolator, lang-mustache, lang-painless, reindex, aggs-matrix-stats, lang-expression, ingest-common, lang-groovy], plugins []
[2016-07-15 20:50:16,608][INFO ][env                      ] [Raa of the Caves] using [1] data paths, mounts [[/ (turtle/_lxd/containers/base-elasticsearch-5-0-0-alpha4)]], net usable_space [8.7tb], net total_space [8.7tb], spins? [possibly], types [zfs]
[2016-07-15 20:50:16,608][INFO ][env                      ] [Raa of the Caves] heap size [1.9gb], compressed ordinary object pointers [true]
[2016-07-15 20:50:18,062][INFO ][node                     ] [Raa of the Caves] initialized
[2016-07-15 20:50:18,062][INFO ][node                     ] [Raa of the Caves] starting ...
[2016-07-15 20:50:18,231][INFO ][transport                ] [Raa of the Caves] publish_address {10.10.1.67:9300}, bound_addresses {10.10.1.67:9300}
Exception in thread "main" java.lang.RuntimeException: bootstrap checks failed
initial heap size [268435456] not equal to maximum heap size [2147483648]; this can cause resize pauses and prevents mlockall from locking the entire heap
please set [discovery.zen.minimum_master_nodes] to a majority of the number of master eligible nodes in your cluster
max virtual memory areas vm.max_map_count [65530] likely too low, increase to at least [262144]
        at org.elasticsearch.bootstrap.BootstrapCheck.check(BootstrapCheck.java:125)
        at org.elasticsearch.bootstrap.BootstrapCheck.check(BootstrapCheck.java:85)
        at org.elasticsearch.bootstrap.BootstrapCheck.check(BootstrapCheck.java:65)
        at org.elasticsearch.bootstrap.Bootstrap$5.validateNodeBeforeAcceptingRequests(Bootstrap.java:178)
        at org.elasticsearch.node.Node.start(Node.java:373)
        at org.elasticsearch.bootstrap.Bootstrap.start(Bootstrap.java:193)
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:252)
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:96)
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:91)
        at org.elasticsearch.cli.SettingCommand.execute(SettingCommand.java:54)
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:91)
        at org.elasticsearch.cli.Command.main(Command.java:53)
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:70)
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:63)
Refer to the log for complete error details.
[2016-07-15 20:50:18,244][INFO ][node                     ] [Raa of the Caves] stopping ...
[2016-07-15 20:50:18,267][INFO ][node                     ] [Raa of the Caves] stopped
[2016-07-15 20:50:18,267][INFO ][node                     ] [Raa of the Caves] closing ...
[2016-07-15 20:50:18,278][INFO ][node                     ] [Raa of the Caves] closed
elasticsearch@base-elasticsearch-5-0-0-alpha4:~$

5.0.0alpha4 vm.max_map_count value is 65530:

root@base-elasticsearch-5-0-0-alpha4:~# sysctl -a | grep vm.max_map_count
sysctl: permission denied on key 'fs.protected_hardlinks'
sysctl: permission denied on key 'fs.protected_symlinks'
sysctl: permission denied on key 'kernel.cad_pid'
sysctl: permission denied on key 'kernel.unprivileged_userns_apparmor_policy'
sysctl: permission denied on key 'kernel.usermodehelper.bset'
sysctl: permission denied on key 'kernel.usermodehelper.inheritable'
sysctl: reading key "net.ipv6.conf.all.stable_secret"
sysctl: reading key "net.ipv6.conf.default.stable_secret"
sysctl: reading key "net.ipv6.conf.eth0.stable_secret"
sysctl: reading key "net.ipv6.conf.lo.stable_secret"
vm.max_map_count = 65530

2.3.4 vm.max_map_count value is 65530:

root@base-elasticsearch-2-3-4:~# sysctl -a | grep vm.max_map_count
sysctl: permission denied on key 'fs.protected_hardlinks'
sysctl: permission denied on key 'fs.protected_symlinks'
sysctl: permission denied on key 'kernel.cad_pid'
sysctl: permission denied on key 'kernel.unprivileged_userns_apparmor_policy'
sysctl: permission denied on key 'kernel.usermodehelper.bset'
sysctl: permission denied on key 'kernel.usermodehelper.inheritable'
sysctl: reading key "net.ipv6.conf.all.stable_secret"
sysctl: reading key "net.ipv6.conf.default.stable_secret"
sysctl: reading key "net.ipv6.conf.eth0.stable_secret"
sysctl: reading key "net.ipv6.conf.lo.stable_secret"
vm.max_map_count = 65530
@nik9000

This comment has been minimized.

Show comment
Hide comment
@nik9000

nik9000 Jul 15, 2016

Contributor

It told you why it isn't starting:

  • initial heap size [268435456] not equal to maximum heap size [2147483648]; this can cause resize pauses and prevents mlockall from locking the entire heap
  • please set [discovery.zen.minimum_master_nodes] to a majority of the number of master eligible nodes in your cluster
  • max virtual memory areas vm.max_map_count [65530] likely too low, increase to at least [262144]

It doesn't check these things if you bind to localhost because it assumes that you aren't running it in production. If you are running in production these are the minimum changes that we recommend.

Contributor

nik9000 commented Jul 15, 2016

It told you why it isn't starting:

  • initial heap size [268435456] not equal to maximum heap size [2147483648]; this can cause resize pauses and prevents mlockall from locking the entire heap
  • please set [discovery.zen.minimum_master_nodes] to a majority of the number of master eligible nodes in your cluster
  • max virtual memory areas vm.max_map_count [65530] likely too low, increase to at least [262144]

It doesn't check these things if you bind to localhost because it assumes that you aren't running it in production. If you are running in production these are the minimum changes that we recommend.

@nik9000 nik9000 closed this Jul 15, 2016

@jasontedor

This comment has been minimized.

Show comment
Hide comment
@jasontedor

jasontedor Jul 15, 2016

Member

Since you're binding to an external interface, the Elasticsearch bootstrap checks are enforced. In particular, you're failing three checks:

initial heap size [268435456] not equal to maximum heap size [2147483648]; this can cause resize pauses and prevents mlockall from locking the entire heap
please set [discovery.zen.minimum_master_nodes] to a majority of the number of master eligible nodes in your cluster
max virtual memory areas vm.max_map_count [65530] likely too low, increase to at least [262144]

These are all documented in the bootstrap check docs.

You'll need to set the initial heap size equal to your desired max heap size, configure discovery.zen.minimum_master_nodes, and set vm.max_map_count to at least 262144.

Member

jasontedor commented Jul 15, 2016

Since you're binding to an external interface, the Elasticsearch bootstrap checks are enforced. In particular, you're failing three checks:

initial heap size [268435456] not equal to maximum heap size [2147483648]; this can cause resize pauses and prevents mlockall from locking the entire heap
please set [discovery.zen.minimum_master_nodes] to a majority of the number of master eligible nodes in your cluster
max virtual memory areas vm.max_map_count [65530] likely too low, increase to at least [262144]

These are all documented in the bootstrap check docs.

You'll need to set the initial heap size equal to your desired max heap size, configure discovery.zen.minimum_master_nodes, and set vm.max_map_count to at least 262144.

@naisanza

This comment has been minimized.

Show comment
Hide comment
@naisanza

naisanza Jul 15, 2016

@nik9000 @jasontedor going through resolving the bootstrap tests one-by-one, with bootstrap.memory_lock: true set in the config, should you still be encountering the initial heap size not equal to maximum heap size error?

I'm still getting that error

naisanza commented Jul 15, 2016

@nik9000 @jasontedor going through resolving the bootstrap tests one-by-one, with bootstrap.memory_lock: true set in the config, should you still be encountering the initial heap size not equal to maximum heap size error?

I'm still getting that error

@jasontedor

This comment has been minimized.

Show comment
Hide comment
@jasontedor

jasontedor Jul 15, 2016

Member

The setting bootstrap.memory_lock has nothing to do with setting the heap size. It's in the docs that I linked to.

Member

jasontedor commented Jul 15, 2016

The setting bootstrap.memory_lock has nothing to do with setting the heap size. It's in the docs that I linked to.

@naisanza

This comment has been minimized.

Show comment
Hide comment
@naisanza

naisanza Jul 16, 2016

@jasontedor that's where I got it from

https://www.elastic.co/guide/en/elasticsearch/reference/master/_heap_size_check.html

Heap size check
If a JVM is started with unequal initial and max heap size, it can be prone to pauses as the JVM heap is resized during system usage. To avoid these resize pauses, it’s best to start the JVM with the initial heap size equal to the maximum heap size. Additionally, if bootstrap.memory_lock is enabled, the JVM will lock the initial size of the heap on startup. If the initial heap size is not equal to the maximum heap size, after a resize it will not be the case that all of the JVM heap is locked in memory. To pass the heap size check, you must configure the heap size.

naisanza commented Jul 16, 2016

@jasontedor that's where I got it from

https://www.elastic.co/guide/en/elasticsearch/reference/master/_heap_size_check.html

Heap size check
If a JVM is started with unequal initial and max heap size, it can be prone to pauses as the JVM heap is resized during system usage. To avoid these resize pauses, it’s best to start the JVM with the initial heap size equal to the maximum heap size. Additionally, if bootstrap.memory_lock is enabled, the JVM will lock the initial size of the heap on startup. If the initial heap size is not equal to the maximum heap size, after a resize it will not be the case that all of the JVM heap is locked in memory. To pass the heap size check, you must configure the heap size.

@jasontedor

This comment has been minimized.

Show comment
Hide comment
@jasontedor

jasontedor Jul 16, 2016

Member

That is telling you the advantage of setting the initial heap size to the desired maximum heap size. If you follow the last link on that page it will take you directly to how to set the heap size:

To pass the heap size check, you must configure the heap size.

Member

jasontedor commented Jul 16, 2016

That is telling you the advantage of setting the initial heap size to the desired maximum heap size. If you follow the last link on that page it will take you directly to how to set the heap size:

To pass the heap size check, you must configure the heap size.

@naisanza

This comment has been minimized.

Show comment
Hide comment
@naisanza

naisanza Jul 30, 2016

Solved the vm.max_map_count lxc/lxd#2206 (comment)

naisanza commented Jul 30, 2016

Solved the vm.max_map_count lxc/lxd#2206 (comment)

@naisanza

This comment has been minimized.

Show comment
Hide comment
@naisanza

naisanza Aug 2, 2016

Working elasticsearch.yml:

cluster.name: beta
network.host: _site_
discovery.zen.minimum_master_nodes: 1

Working elasticsearch launch parameters:
ES_JAVA_OPTS='-Xms2048m -Xmx2048m' elasticsearch

It works now

naisanza commented Aug 2, 2016

Working elasticsearch.yml:

cluster.name: beta
network.host: _site_
discovery.zen.minimum_master_nodes: 1

Working elasticsearch launch parameters:
ES_JAVA_OPTS='-Xms2048m -Xmx2048m' elasticsearch

It works now

@naisanza naisanza changed the title from Vanilla Elasticsearch 5.0.0alpha4 unable to start to Vanilla Elasticsearch 5.0.0alpha4 unable to start in LXD container Aug 2, 2016

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment