Skip to content
This repository has been archived by the owner on Jun 11, 2021. It is now read-only.

ElasticSearch 7+ "Native controller process has stopped - no new native processes can be started" #5029

Open
ediewoelfle opened this issue Jul 2, 2019 · 6 comments
Open

ElasticSearch 7+ "Native controller process has stopped - no new native processes can be started" #5029

ediewoelfle opened this issue Jul 2, 2019 · 6 comments

Comments

@ediewoelfle
Copy link

Expected behavior

Logs from a new 6.8.0 ES container:

OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
[2019-07-02T17:33:16,932][INFO ][o.e.e.NodeEnvironment    ] [vUjaJui] using [1] data paths, mounts [[/ (overlay)]], net usable_space [42.4gb], net total_space [58.4gb], types [overlay]
[2019-07-02T17:33:16,938][INFO ][o.e.e.NodeEnvironment    ] [vUjaJui] heap size [990.7mb], compressed ordinary object pointers [true]
[2019-07-02T17:33:16,941][INFO ][o.e.n.Node               ] [vUjaJui] node name derived from node ID [vUjaJui4QOShI_nBMJeJ2Q]; set [node.name] to override
[2019-07-02T17:33:16,942][INFO ][o.e.n.Node               ] [vUjaJui] version[6.8.0], pid[1], build[default/docker/65b6179/2019-05-15T20:06:13.172855Z], OS[Linux/4.9.125-linuxkit/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/12.0.1/12.0.1+12]
[2019-07-02T17:33:16,942][INFO ][o.e.n.Node               ] [vUjaJui] JVM arguments [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/tmp/elasticsearch-3950135291909166212, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Djava.locale.providers=COMPAT, -XX:UseAVX=2, -Des.cgroups.hierarchy.override=/, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/usr/share/elasticsearch/config, -Des.distribution.flavor=default, -Des.distribution.type=docker]
[2019-07-02T17:33:19,182][INFO ][o.e.p.PluginsService     ] [vUjaJui] loaded module [aggs-matrix-stats]
[2019-07-02T17:33:19,182][INFO ][o.e.p.PluginsService     ] [vUjaJui] loaded module [analysis-common]
[2019-07-02T17:33:19,182][INFO ][o.e.p.PluginsService     ] [vUjaJui] loaded module [ingest-common]
[2019-07-02T17:33:19,182][INFO ][o.e.p.PluginsService     ] [vUjaJui] loaded module [ingest-geoip]
[2019-07-02T17:33:19,182][INFO ][o.e.p.PluginsService     ] [vUjaJui] loaded module [ingest-user-agent]
[2019-07-02T17:33:19,183][INFO ][o.e.p.PluginsService     ] [vUjaJui] loaded module [lang-expression]
[2019-07-02T17:33:19,183][INFO ][o.e.p.PluginsService     ] [vUjaJui] loaded module [lang-mustache]
[2019-07-02T17:33:19,183][INFO ][o.e.p.PluginsService     ] [vUjaJui] loaded module [lang-painless]
[2019-07-02T17:33:19,183][INFO ][o.e.p.PluginsService     ] [vUjaJui] loaded module [mapper-extras]
[2019-07-02T17:33:19,184][INFO ][o.e.p.PluginsService     ] [vUjaJui] loaded module [parent-join]
[2019-07-02T17:33:19,184][INFO ][o.e.p.PluginsService     ] [vUjaJui] loaded module [percolator]
[2019-07-02T17:33:19,185][INFO ][o.e.p.PluginsService     ] [vUjaJui] loaded module [rank-eval]
[2019-07-02T17:33:19,185][INFO ][o.e.p.PluginsService     ] [vUjaJui] loaded module [reindex]
[2019-07-02T17:33:19,186][INFO ][o.e.p.PluginsService     ] [vUjaJui] loaded module [repository-url]
[2019-07-02T17:33:19,187][INFO ][o.e.p.PluginsService     ] [vUjaJui] loaded module [transport-netty4]
[2019-07-02T17:33:19,187][INFO ][o.e.p.PluginsService     ] [vUjaJui] loaded module [tribe]
[2019-07-02T17:33:19,187][INFO ][o.e.p.PluginsService     ] [vUjaJui] loaded module [x-pack-ccr]
[2019-07-02T17:33:19,188][INFO ][o.e.p.PluginsService     ] [vUjaJui] loaded module [x-pack-core]
[2019-07-02T17:33:19,188][INFO ][o.e.p.PluginsService     ] [vUjaJui] loaded module [x-pack-deprecation]
[2019-07-02T17:33:19,188][INFO ][o.e.p.PluginsService     ] [vUjaJui] loaded module [x-pack-graph]
[2019-07-02T17:33:19,189][INFO ][o.e.p.PluginsService     ] [vUjaJui] loaded module [x-pack-ilm]
[2019-07-02T17:33:19,189][INFO ][o.e.p.PluginsService     ] [vUjaJui] loaded module [x-pack-logstash]
[2019-07-02T17:33:19,189][INFO ][o.e.p.PluginsService     ] [vUjaJui] loaded module [x-pack-ml]
[2019-07-02T17:33:19,190][INFO ][o.e.p.PluginsService     ] [vUjaJui] loaded module [x-pack-monitoring]
[2019-07-02T17:33:19,190][INFO ][o.e.p.PluginsService     ] [vUjaJui] loaded module [x-pack-rollup]
[2019-07-02T17:33:19,190][INFO ][o.e.p.PluginsService     ] [vUjaJui] loaded module [x-pack-security]
[2019-07-02T17:33:19,191][INFO ][o.e.p.PluginsService     ] [vUjaJui] loaded module [x-pack-sql]
[2019-07-02T17:33:19,191][INFO ][o.e.p.PluginsService     ] [vUjaJui] loaded module [x-pack-upgrade]
[2019-07-02T17:33:19,191][INFO ][o.e.p.PluginsService     ] [vUjaJui] loaded module [x-pack-watcher]
[2019-07-02T17:33:19,192][INFO ][o.e.p.PluginsService     ] [vUjaJui] no plugins loaded
[2019-07-02T17:33:23,903][INFO ][o.e.x.s.a.s.FileRolesStore] [vUjaJui] parsed [0] roles from file [/usr/share/elasticsearch/config/roles.yml]
[2019-07-02T17:33:24,812][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [vUjaJui] [controller/87] [Main.cc@109] controller (64 bit): Version 6.8.0 (Build e6cf25e2acc5ec) Copyright (c) 2019 Elasticsearch BV
[2019-07-02T17:33:26,105][INFO ][o.e.d.DiscoveryModule    ] [vUjaJui] using discovery type [zen] and host providers [settings]
[2019-07-02T17:33:27,495][INFO ][o.e.n.Node               ] [vUjaJui] initialized
[2019-07-02T17:33:27,495][INFO ][o.e.n.Node               ] [vUjaJui] starting ...
[2019-07-02T17:33:27,754][INFO ][o.e.t.TransportService   ] [vUjaJui] publish_address {172.17.0.4:9300}, bound_addresses {0.0.0.0:9300}
[2019-07-02T17:33:27,780][INFO ][o.e.b.BootstrapChecks    ] [vUjaJui] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2019-07-02T17:33:30,876][INFO ][o.e.c.s.MasterService    ] [vUjaJui] zen-disco-elected-as-master ([0] nodes joined), reason: new_master {vUjaJui}{vUjaJui4QOShI_nBMJeJ2Q}{77LyFw04RfSvm5LzMZLTOA}{172.17.0.4}{172.17.0.4:9300}{ml.machine_memory=2095869952, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}
[2019-07-02T17:33:30,891][INFO ][o.e.c.s.ClusterApplierService] [vUjaJui] new_master {vUjaJui}{vUjaJui4QOShI_nBMJeJ2Q}{77LyFw04RfSvm5LzMZLTOA}{172.17.0.4}{172.17.0.4:9300}{ml.machine_memory=2095869952, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}, reason: apply cluster state (from master [master {vUjaJui}{vUjaJui4QOShI_nBMJeJ2Q}{77LyFw04RfSvm5LzMZLTOA}{172.17.0.4}{172.17.0.4:9300}{ml.machine_memory=2095869952, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true} committed version [1] source [zen-disco-elected-as-master ([0] nodes joined)]])
[2019-07-02T17:33:30,981][INFO ][o.e.h.n.Netty4HttpServerTransport] [vUjaJui] publish_address {172.17.0.4:9200}, bound_addresses {0.0.0.0:9200}
[2019-07-02T17:33:30,982][INFO ][o.e.n.Node               ] [vUjaJui] started
[2019-07-02T17:33:31,041][WARN ][o.e.x.s.a.s.m.NativeRoleMappingStore] [vUjaJui] Failed to clear cache for realms [[]]
[2019-07-02T17:33:31,184][INFO ][o.e.g.GatewayService     ] [vUjaJui] recovered [0] indices into cluster_state
[2019-07-02T17:33:31,422][INFO ][o.e.c.m.MetaDataIndexTemplateService] [vUjaJui] adding template [.watch-history-9] for index patterns [.watcher-history-9*]
[2019-07-02T17:33:31,456][INFO ][o.e.c.m.MetaDataIndexTemplateService] [vUjaJui] adding template [.watches] for index patterns [.watches*]
[2019-07-02T17:33:31,486][INFO ][o.e.c.m.MetaDataIndexTemplateService] [vUjaJui] adding template [.triggered_watches] for index patterns [.triggered_watches*]
[2019-07-02T17:33:31,531][INFO ][o.e.c.m.MetaDataIndexTemplateService] [vUjaJui] adding template [.monitoring-logstash] for index patterns [.monitoring-logstash-6-*]
[2019-07-02T17:33:31,587][INFO ][o.e.c.m.MetaDataIndexTemplateService] [vUjaJui] adding template [.monitoring-es] for index patterns [.monitoring-es-6-*]
[2019-07-02T17:33:31,620][INFO ][o.e.c.m.MetaDataIndexTemplateService] [vUjaJui] adding template [.monitoring-alerts] for index patterns [.monitoring-alerts-6]
[2019-07-02T17:33:31,665][INFO ][o.e.c.m.MetaDataIndexTemplateService] [vUjaJui] adding template [.monitoring-beats] for index patterns [.monitoring-beats-6-*]
[2019-07-02T17:33:31,702][INFO ][o.e.c.m.MetaDataIndexTemplateService] [vUjaJui] adding template [.monitoring-kibana] for index patterns [.monitoring-kibana-6-*]
[2019-07-02T17:33:31,795][INFO ][o.e.l.LicenseService     ] [vUjaJui] license [d3424e40-0481-4654-be74-7ebdc85f903b] mode [basic] - valid

Actual behavior

Logs from a new 7.2.0 ES container:

OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
{"type": "server", "timestamp": "2019-07-02T17:47:05,402+0000", "level": "INFO", "component": "o.e.e.NodeEnvironment", "cluster.name": "docker-cluster", "node.name": "74175a1d24d7",  "message": "using [1] data paths, mounts [[/ (overlay)]], net usable_space [42.4gb], net total_space [58.4gb], types [overlay]"  }
{"type": "server", "timestamp": "2019-07-02T17:47:05,408+0000", "level": "INFO", "component": "o.e.e.NodeEnvironment", "cluster.name": "docker-cluster", "node.name": "74175a1d24d7",  "message": "heap size [990.7mb], compressed ordinary object pointers [true]"  }
{"type": "server", "timestamp": "2019-07-02T17:47:05,412+0000", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "docker-cluster", "node.name": "74175a1d24d7",  "message": "node name [74175a1d24d7], node ID [7sc8J0dCRJWCC6O2m0EmEw], cluster name [docker-cluster]"  }
{"type": "server", "timestamp": "2019-07-02T17:47:05,413+0000", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "docker-cluster", "node.name": "74175a1d24d7",  "message": "version[7.2.0], pid[1], build[default/docker/508c38a/2019-06-20T15:54:18.811730Z], OS[Linux/4.9.125-linuxkit/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/12.0.1/12.0.1+12]"  }
{"type": "server", "timestamp": "2019-07-02T17:47:05,414+0000", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "docker-cluster", "node.name": "74175a1d24d7",  "message": "JVM home [/usr/share/elasticsearch/jdk]"  }
{"type": "server", "timestamp": "2019-07-02T17:47:05,415+0000", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "docker-cluster", "node.name": "74175a1d24d7",  "message": "JVM arguments [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/tmp/elasticsearch-5100568283784166773, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Djava.locale.providers=COMPAT, -Des.cgroups.hierarchy.override=/, -Dio.netty.allocator.type=unpooled, -XX:MaxDirectMemorySize=536870912, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/usr/share/elasticsearch/config, -Des.distribution.flavor=default, -Des.distribution.type=docker, -Des.bundled_jdk=true]"  }
{"type": "server", "timestamp": "2019-07-02T17:47:07,884+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "74175a1d24d7",  "message": "loaded module [aggs-matrix-stats]"  }
{"type": "server", "timestamp": "2019-07-02T17:47:07,884+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "74175a1d24d7",  "message": "loaded module [analysis-common]"  }
{"type": "server", "timestamp": "2019-07-02T17:47:07,885+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "74175a1d24d7",  "message": "loaded module [data-frame]"  }
{"type": "server", "timestamp": "2019-07-02T17:47:07,886+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "74175a1d24d7",  "message": "loaded module [ingest-common]"  }
{"type": "server", "timestamp": "2019-07-02T17:47:07,888+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "74175a1d24d7",  "message": "loaded module [ingest-geoip]"  }
{"type": "server", "timestamp": "2019-07-02T17:47:07,889+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "74175a1d24d7",  "message": "loaded module [ingest-user-agent]"  }
{"type": "server", "timestamp": "2019-07-02T17:47:07,890+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "74175a1d24d7",  "message": "loaded module [lang-expression]"  }
{"type": "server", "timestamp": "2019-07-02T17:47:07,891+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "74175a1d24d7",  "message": "loaded module [lang-mustache]"  }
{"type": "server", "timestamp": "2019-07-02T17:47:07,892+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "74175a1d24d7",  "message": "loaded module [lang-painless]"  }
{"type": "server", "timestamp": "2019-07-02T17:47:07,893+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "74175a1d24d7",  "message": "loaded module [mapper-extras]"  }
{"type": "server", "timestamp": "2019-07-02T17:47:07,894+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "74175a1d24d7",  "message": "loaded module [parent-join]"  }
{"type": "server", "timestamp": "2019-07-02T17:47:07,894+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "74175a1d24d7",  "message": "loaded module [percolator]"  }
{"type": "server", "timestamp": "2019-07-02T17:47:07,895+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "74175a1d24d7",  "message": "loaded module [rank-eval]"  }
{"type": "server", "timestamp": "2019-07-02T17:47:07,896+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "74175a1d24d7",  "message": "loaded module [reindex]"  }
{"type": "server", "timestamp": "2019-07-02T17:47:07,896+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "74175a1d24d7",  "message": "loaded module [repository-url]"  }
{"type": "server", "timestamp": "2019-07-02T17:47:07,897+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "74175a1d24d7",  "message": "loaded module [transport-netty4]"  }
{"type": "server", "timestamp": "2019-07-02T17:47:07,897+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "74175a1d24d7",  "message": "loaded module [x-pack-ccr]"  }
{"type": "server", "timestamp": "2019-07-02T17:47:07,898+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "74175a1d24d7",  "message": "loaded module [x-pack-core]"  }
{"type": "server", "timestamp": "2019-07-02T17:47:07,898+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "74175a1d24d7",  "message": "loaded module [x-pack-deprecation]"  }
{"type": "server", "timestamp": "2019-07-02T17:47:07,899+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "74175a1d24d7",  "message": "loaded module [x-pack-graph]"  }
{"type": "server", "timestamp": "2019-07-02T17:47:07,899+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "74175a1d24d7",  "message": "loaded module [x-pack-ilm]"  }
{"type": "server", "timestamp": "2019-07-02T17:47:07,899+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "74175a1d24d7",  "message": "loaded module [x-pack-logstash]"  }
{"type": "server", "timestamp": "2019-07-02T17:47:07,900+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "74175a1d24d7",  "message": "loaded module [x-pack-ml]"  }
{"type": "server", "timestamp": "2019-07-02T17:47:07,900+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "74175a1d24d7",  "message": "loaded module [x-pack-monitoring]"  }
{"type": "server", "timestamp": "2019-07-02T17:47:07,901+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "74175a1d24d7",  "message": "loaded module [x-pack-rollup]"  }
{"type": "server", "timestamp": "2019-07-02T17:47:07,901+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "74175a1d24d7",  "message": "loaded module [x-pack-security]"  }
{"type": "server", "timestamp": "2019-07-02T17:47:07,901+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "74175a1d24d7",  "message": "loaded module [x-pack-sql]"  }
{"type": "server", "timestamp": "2019-07-02T17:47:07,902+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "74175a1d24d7",  "message": "loaded module [x-pack-watcher]"  }
{"type": "server", "timestamp": "2019-07-02T17:47:07,903+0000", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "docker-cluster", "node.name": "74175a1d24d7",  "message": "no plugins loaded"  }
{"type": "server", "timestamp": "2019-07-02T17:47:12,944+0000", "level": "INFO", "component": "o.e.x.s.a.s.FileRolesStore", "cluster.name": "docker-cluster", "node.name": "74175a1d24d7",  "message": "parsed [0] roles from file [/usr/share/elasticsearch/config/roles.yml]"  }
{"type": "server", "timestamp": "2019-07-02T17:47:13,950+0000", "level": "INFO", "component": "o.e.x.m.p.l.CppLogMessageHandler", "cluster.name": "docker-cluster", "node.name": "74175a1d24d7",  "message": "[controller/103] [Main.cc@110] controller (64 bit): Version 7.2.0 (Build 65aefcbfce449b) Copyright (c) 2019 Elasticsearch BV"  }
{"type": "server", "timestamp": "2019-07-02T17:47:14,782+0000", "level": "DEBUG", "component": "o.e.a.ActionModule", "cluster.name": "docker-cluster", "node.name": "74175a1d24d7",  "message": "Using REST wrapper from plugin org.elasticsearch.xpack.security.Security"  }
{"type": "server", "timestamp": "2019-07-02T17:47:15,268+0000", "level": "INFO", "component": "o.e.d.DiscoveryModule", "cluster.name": "docker-cluster", "node.name": "74175a1d24d7",  "message": "using discovery type [zen] and seed hosts providers [settings]"  }
{"type": "server", "timestamp": "2019-07-02T17:47:16,522+0000", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "docker-cluster", "node.name": "74175a1d24d7",  "message": "initialized"  }
{"type": "server", "timestamp": "2019-07-02T17:47:16,523+0000", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "docker-cluster", "node.name": "74175a1d24d7",  "message": "starting ..."  }
{"type": "server", "timestamp": "2019-07-02T17:47:16,821+0000", "level": "INFO", "component": "o.e.t.TransportService", "cluster.name": "docker-cluster", "node.name": "74175a1d24d7",  "message": "publish_address {172.17.0.4:9300}, bound_addresses {0.0.0.0:9300}"  }
{"type": "server", "timestamp": "2019-07-02T17:47:16,840+0000", "level": "INFO", "component": "o.e.b.BootstrapChecks", "cluster.name": "docker-cluster", "node.name": "74175a1d24d7",  "message": "bound or publishing to a non-loopback address, enforcing bootstrap checks"  }
ERROR: [1] bootstrap checks failed
[1]: the default discovery settings are unsuitable for production use; at least one of [discovery.seed_hosts, discovery.seed_providers, cluster.initial_master_nodes] must be configured
{"type": "server", "timestamp": "2019-07-02T17:47:16,861+0000", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "docker-cluster", "node.name": "74175a1d24d7",  "message": "stopping ..."  }
{"type": "server", "timestamp": "2019-07-02T17:47:16,922+0000", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "docker-cluster", "node.name": "74175a1d24d7",  "message": "stopped"  }
{"type": "server", "timestamp": "2019-07-02T17:47:16,923+0000", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "docker-cluster", "node.name": "74175a1d24d7",  "message": "closing ..."  }
{"type": "server", "timestamp": "2019-07-02T17:47:16,953+0000", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "docker-cluster", "node.name": "74175a1d24d7",  "message": "closed"  }
{"type": "server", "timestamp": "2019-07-02T17:47:16,959+0000", "level": "INFO", "component": "o.e.x.m.p.NativeController", "cluster.name": "docker-cluster", "node.name": "74175a1d24d7",  "message": "Native controller process has stopped - no new native processes can be started"  }

Information about the Issue

Every ES container above 6.8.0 fails to start.

Steps to reproduce the behavior

  1. Create a new ES container with a 7.0.0 tag or above.
@ediewoelfle
Copy link
Author

adding discovery.type = single-node to the environment variables fixes the issue, but undesirable.

@thangchung
Copy link

@ediewoelfle Thank you. You save my day 👍

@lupguo
Copy link

lupguo commented Nov 14, 2019

My environment is initially equipped with ES cluster, and one node always exits, and then reports exit with 137 , the query is a memory problem.

Therefore, I switched to a single ES node and modified my docker-compose.yml configuration. I modified the environment configuration as discovery.type=single-node , and found error report elastic_es01_1 exited with code 1!

My fix: Remove the previous volume binding directory directly(ES Cluster), and run docker again and everything is ok!

My docker-compose.yml:

version: "3.7"
services:
    es01:
        image: elasticsearch:7.4.2
        environment:
            - node.name=es01
            - cluster.name=es-docker-cluster
            - discovery.type=single-node
        volumes:
            - /data/elastic/data01:/usr/share/elasticsearch/data
        networks:
            - elastic
        ports:
            - 9200:9200
            - 9300:9300
networks:
  elastic:
    driver: bridge

@Sysa
Copy link

Sysa commented Nov 22, 2019

Looks like you are entered into production mode after explicit network.host: 0.0.0.0 and then, according to your logs, it fails on bootstrap checks :

[1]: the default discovery settings are unsuitable for production use; at least one of [discovery.seed_hosts, discovery.seed_providers, cluster.initial_master_nodes] must be configured

It is required to do proper configuration setup since 7.0+ version.
You need to specify at least one of discovery.seed_hosts, discovery.seed_providers or cluster.initial_master_nodes in $ES_HOME/config/elasticsearch.yml

@IntrovertedFL
Copy link

IntrovertedFL commented Feb 23, 2020 •
edited
Loading

Thank you

@slrbl slrbl mentioned this issue Apr 15, 2020
Closed
@amitrangari
Copy link

My environment is initially equipped with ES cluster, and one node always exits, and then reports exit with 137 , the query is a memory problem.

Therefore, I switched to a single ES node and modified my docker-compose.yml configuration. I modified the environment configuration as discovery.type=single-node , and found error report elastic_es01_1 exited with code 1!

My fix: Remove the previous volume binding directory directly(ES Cluster), and run docker again and everything is ok!

My docker-compose.yml:

version: "3.7"
services:
    es01:
        image: elasticsearch:7.4.2
        environment:
            - node.name=es01
            - cluster.name=es-docker-cluster
            - discovery.type=single-node
        volumes:
            - /data/elastic/data01:/usr/share/elasticsearch/data
        networks:
            - elastic
        ports:
            - 9200:9200
            - 9300:9300
networks:
  elastic:
    driver: bridge

This helped me

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@thangchung @Sysa @ediewoelfle @amitrangari @lupguo @IntrovertedFL