Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

DataNode is not starting up after provisioning certificates #41

Closed
danielescalona opened this issue Nov 15, 2023 · 14 comments
Closed

DataNode is not starting up after provisioning certificates #41

danielescalona opened this issue Nov 15, 2023 · 14 comments
Assignees
@janheise
Labels
triaged

Comments

@danielescalona
Copy link

danielescalona commented Nov 15, 2023 •
edited by kroepke
Loading

I am getting this on the log under datanode. can someone help?

2023-11-15T19:16:38.201Z WARN  [OpensearchProcessImpl] WARNING: System::setSecurityManager will be removed in a future release
2023-11-15T19:16:39.376Z WARN  [OpensearchNodeHeartbeat] Opensearch REST api of process 1058 unavailable. Cause: Connection refused
2023-11-15T19:16:39.813Z WARN  [OpensearchProcessImpl] WARNING: A terminally deprecated method in java.lang.System has been called
2023-11-15T19:16:39.813Z WARN  [OpensearchProcessImpl] WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.Security (file:/usr/share/graylog-datanode/dist/opensearch-2.10.0-linux-x64/lib/opensearch-2.10.0.jar)
2023-11-15T19:16:39.814Z WARN  [OpensearchProcessImpl] WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.Security
2023-11-15T19:16:39.814Z WARN  [OpensearchProcessImpl] WARNING: System::setSecurityManager will be removed in a future release
2023-11-15T19:16:39.844Z INFO  [OpensearchProcessImpl] [2023-11-15T19:16:39,840][INFO ][o.o.n.Node               ] [0eace0b2164c] version[2.10.0], pid[1058], build[tar/eee49cb340edc6c4d489bcd9324dda571fc8dc03/2023-09-20T23:54:29.889267151Z], OS[Linux/6.1.8-0-lts/amd64], JVM[Eclipse Adoptium/OpenJDK 64-Bit Server VM/17.0.8/17.0.8+7]
2023-11-15T19:16:39.846Z INFO  [OpensearchProcessImpl] [2023-11-15T19:16:39,845][INFO ][o.o.n.Node               ] [0eace0b2164c] JVM home [/usr/share/graylog-datanode/dist/opensearch-2.10.0-linux-x64/jdk], using bundled JDK/JRE [true]
2023-11-15T19:16:39.847Z INFO  [OpensearchProcessImpl] [2023-11-15T19:16:39,846][INFO ][o.o.n.Node               ] [0eace0b2164c] JVM arguments [-Xshare:auto, -Dopensearch.networkaddress.cache.ttl=60, -Dopensearch.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Xms1g, -Xmx1g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/tmp/opensearch-2544192809827378494, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=/tmp/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/tmp/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -XX:MaxDirectMemorySize=536870912, -Dopensearch.path.home=/usr/share/graylog-datanode/dist/opensearch-2.10.0-linux-x64, -Dopensearch.path.conf=/var/lib/graylog-datanode/opensearch/config/8b2841e7-04e2-4ed2-9216-ed0e4e75f44b/opensearch, -Dopensearch.distribution.type=tar, -Dopensearch.bundled_jdk=true]
2023-11-15T19:16:41.224Z INFO  [OpensearchProcessImpl] [2023-11-15T19:16:41,223][INFO ][o.o.s.s.t.SSLConfig      ] [0eace0b2164c] SSL dual mode is disabled
2023-11-15T19:16:41.224Z INFO  [OpensearchProcessImpl] [2023-11-15T19:16:41,224][INFO ][o.o.s.OpenSearchSecurityPlugin] [0eace0b2164c] OpenSearch Config path is /var/lib/graylog-datanode/opensearch/config/8b2841e7-04e2-4ed2-9216-ed0e4e75f44b/opensearch
2023-11-15T19:16:41.486Z INFO  [OpensearchProcessImpl] [2023-11-15T19:16:41,485][INFO ][o.o.s.s.DefaultSecurityKeyStore] [0eace0b2164c] JVM supports TLSv1.3
2023-11-15T19:16:41.488Z INFO  [OpensearchProcessImpl] [2023-11-15T19:16:41,488][INFO ][o.o.s.s.DefaultSecurityKeyStore] [0eace0b2164c] Config directory is /var/lib/graylog-datanode/opensearch/config/8b2841e7-04e2-4ed2-9216-ed0e4e75f44b/opensearch/, from there the key- and truststore files are resolved relatively
2023-11-15T19:16:41.491Z INFO  [OpensearchProcessImpl] [2023-11-15T19:16:41,491][WARN ][o.o.s.s.SecureSSLSettings] [0eace0b2164c] Setting [plugins.security.ssl.transport.keystore_password] has a secure counterpart [plugins.security.ssl.transport.keystore_password_secure] which should be used instead - allowing for legacy SSL setups
2023-11-15T19:16:41.498Z INFO  [OpensearchProcessImpl] [2023-11-15T19:16:41,497][WARN ][o.o.s.s.SecureSSLSettings] [0eace0b2164c] Setting [plugins.security.ssl.transport.truststore_password] has a secure counterpart [plugins.security.ssl.transport.truststore_password_secure] which should be used instead - allowing for legacy SSL setups
2023-11-15T19:16:41.651Z INFO  [OpensearchProcessImpl] [2023-11-15T19:16:41,650][WARN ][o.o.s.s.u.SSLCertificateHelper] [0eace0b2164c] Certificate chain for alias datanode contains a root certificate
2023-11-15T19:16:41.846Z INFO  [OpensearchProcessImpl] [2023-11-15T19:16:41,846][WARN ][o.o.s.s.SecureSSLSettings] [0eace0b2164c] Setting [plugins.security.ssl.http.keystore_password] has a secure counterpart [plugins.security.ssl.http.keystore_password_secure] which should be used instead - allowing for legacy SSL setups
2023-11-15T19:16:41.848Z INFO  [OpensearchProcessImpl] [2023-11-15T19:16:41,847][INFO ][o.o.s.s.DefaultSecurityKeyStore] [0eace0b2164c] HTTPS client auth mode OPTIONAL
2023-11-15T19:16:41.874Z INFO  [OpensearchProcessImpl] [2023-11-15T19:16:41,874][WARN ][o.o.s.s.u.SSLCertificateHelper] [0eace0b2164c] Certificate chain for alias datanode contains a root certificate
2023-11-15T19:16:41.895Z INFO  [OpensearchProcessImpl] [2023-11-15T19:16:41,894][WARN ][o.o.s.s.SecureSSLSettings] [0eace0b2164c] Setting [plugins.security.ssl.http.truststore_password] has a secure counterpart [plugins.security.ssl.http.truststore_password_secure] which should be used instead - allowing for legacy SSL setups
2023-11-15T19:16:41.960Z INFO  [OpensearchProcessImpl] [2023-11-15T19:16:41,960][INFO ][o.o.s.s.DefaultSecurityKeyStore] [0eace0b2164c] TLS Transport Client Provider : JDK
2023-11-15T19:16:41.961Z INFO  [OpensearchProcessImpl] [2023-11-15T19:16:41,960][INFO ][o.o.s.s.DefaultSecurityKeyStore] [0eace0b2164c] TLS Transport Server Provider : JDK
2023-11-15T19:16:41.961Z INFO  [OpensearchProcessImpl] [2023-11-15T19:16:41,960][INFO ][o.o.s.s.DefaultSecurityKeyStore] [0eace0b2164c] TLS HTTP Provider             : JDK
2023-11-15T19:16:41.961Z INFO  [OpensearchProcessImpl] [2023-11-15T19:16:41,961][INFO ][o.o.s.s.DefaultSecurityKeyStore] [0eace0b2164c] Enabled TLS protocols for transport layer : [TLSv1.3, TLSv1.2]
2023-11-15T19:16:41.961Z INFO  [OpensearchProcessImpl] [2023-11-15T19:16:41,961][INFO ][o.o.s.s.DefaultSecurityKeyStore] [0eace0b2164c] Enabled TLS protocols for HTTP layer      : [TLSv1.3, TLSv1.2]
2023-11-15T19:16:42.194Z INFO  [OpensearchProcessImpl] [2023-11-15T19:16:42,194][INFO ][o.o.s.OpenSearchSecurityPlugin] [0eace0b2164c] Clustername: datanode-cluster
2023-11-15T19:16:42.589Z INFO  [OpensearchProcessImpl] [2023-11-15T19:16:42,589][INFO ][o.o.i.r.ReindexPlugin    ] [0eace0b2164c] ReindexPlugin reloadSPI called
2023-11-15T19:16:42.590Z INFO  [OpensearchProcessImpl] [2023-11-15T19:16:42,590][INFO ][o.o.i.r.ReindexPlugin    ] [0eace0b2164c] Unable to find any implementation for RemoteReindexExtension
2023-11-15T19:16:42.603Z INFO  [OpensearchProcessImpl] [2023-11-15T19:16:42,603][INFO ][o.o.j.JobSchedulerPlugin ] [0eace0b2164c] Loaded scheduler extension: opendistro_anomaly_detector, index: .opendistro-anomaly-detector-jobs
2023-11-15T19:16:42.607Z INFO  [OpensearchProcessImpl] [2023-11-15T19:16:42,607][INFO ][o.o.p.PluginsService     ] [0eace0b2164c] loaded module [aggs-matrix-stats]
2023-11-15T19:16:42.607Z INFO  [OpensearchProcessImpl] [2023-11-15T19:16:42,607][INFO ][o.o.p.PluginsService     ] [0eace0b2164c] loaded module [analysis-common]
2023-11-15T19:16:42.608Z INFO  [OpensearchProcessImpl] [2023-11-15T19:16:42,607][INFO ][o.o.p.PluginsService     ] [0eace0b2164c] loaded module [geo]
2023-11-15T19:16:42.608Z INFO  [OpensearchProcessImpl] [2023-11-15T19:16:42,607][INFO ][o.o.p.PluginsService     ] [0eace0b2164c] loaded module [ingest-common]
2023-11-15T19:16:42.608Z INFO  [OpensearchProcessImpl] [2023-11-15T19:16:42,608][INFO ][o.o.p.PluginsService     ] [0eace0b2164c] loaded module [ingest-geoip]
2023-11-15T19:16:42.609Z INFO  [OpensearchProcessImpl] [2023-11-15T19:16:42,608][INFO ][o.o.p.PluginsService     ] [0eace0b2164c] loaded module [ingest-user-agent]
2023-11-15T19:16:42.609Z INFO  [OpensearchProcessImpl] [2023-11-15T19:16:42,608][INFO ][o.o.p.PluginsService     ] [0eace0b2164c] loaded module [lang-expression]
2023-11-15T19:16:42.609Z INFO  [OpensearchProcessImpl] [2023-11-15T19:16:42,608][INFO ][o.o.p.PluginsService     ] [0eace0b2164c] loaded module [lang-mustache]
2023-11-15T19:16:42.610Z INFO  [OpensearchProcessImpl] [2023-11-15T19:16:42,609][INFO ][o.o.p.PluginsService     ] [0eace0b2164c] loaded module [lang-painless]
2023-11-15T19:16:42.610Z INFO  [OpensearchProcessImpl] [2023-11-15T19:16:42,609][INFO ][o.o.p.PluginsService     ] [0eace0b2164c] loaded module [mapper-extras]
2023-11-15T19:16:42.611Z INFO  [OpensearchProcessImpl] [2023-11-15T19:16:42,609][INFO ][o.o.p.PluginsService     ] [0eace0b2164c] loaded module [opensearch-dashboards]
2023-11-15T19:16:42.611Z INFO  [OpensearchProcessImpl] [2023-11-15T19:16:42,610][INFO ][o.o.p.PluginsService     ] [0eace0b2164c] loaded module [parent-join]
2023-11-15T19:16:42.612Z INFO  [OpensearchProcessImpl] [2023-11-15T19:16:42,610][INFO ][o.o.p.PluginsService     ] [0eace0b2164c] loaded module [percolator]
2023-11-15T19:16:42.612Z INFO  [OpensearchProcessImpl] [2023-11-15T19:16:42,610][INFO ][o.o.p.PluginsService     ] [0eace0b2164c] loaded module [rank-eval]
2023-11-15T19:16:42.614Z INFO  [OpensearchProcessImpl] [2023-11-15T19:16:42,610][INFO ][o.o.p.PluginsService     ] [0eace0b2164c] loaded module [reindex]
2023-11-15T19:16:42.616Z INFO  [OpensearchProcessImpl] [2023-11-15T19:16:42,610][INFO ][o.o.p.PluginsService     ] [0eace0b2164c] loaded module [repository-url]
2023-11-15T19:16:42.616Z INFO  [OpensearchProcessImpl] [2023-11-15T19:16:42,611][INFO ][o.o.p.PluginsService     ] [0eace0b2164c] loaded module [search-pipeline-common]
2023-11-15T19:16:42.617Z INFO  [OpensearchProcessImpl] [2023-11-15T19:16:42,611][INFO ][o.o.p.PluginsService     ] [0eace0b2164c] loaded module [systemd]
2023-11-15T19:16:42.617Z INFO  [OpensearchProcessImpl] [2023-11-15T19:16:42,611][INFO ][o.o.p.PluginsService     ] [0eace0b2164c] loaded module [transport-netty4]
2023-11-15T19:16:42.618Z INFO  [OpensearchProcessImpl] [2023-11-15T19:16:42,612][INFO ][o.o.p.PluginsService     ] [0eace0b2164c] loaded plugin [opensearch-anomaly-detection]
2023-11-15T19:16:42.618Z INFO  [OpensearchProcessImpl] [2023-11-15T19:16:42,612][INFO ][o.o.p.PluginsService     ] [0eace0b2164c] loaded plugin [opensearch-asynchronous-search]
2023-11-15T19:16:42.619Z INFO  [OpensearchProcessImpl] [2023-11-15T19:16:42,612][INFO ][o.o.p.PluginsService     ] [0eace0b2164c] loaded plugin [opensearch-cross-cluster-replication]
2023-11-15T19:16:42.619Z INFO  [OpensearchProcessImpl] [2023-11-15T19:16:42,613][INFO ][o.o.p.PluginsService     ] [0eace0b2164c] loaded plugin [opensearch-job-scheduler]
2023-11-15T19:16:42.620Z INFO  [OpensearchProcessImpl] [2023-11-15T19:16:42,613][INFO ][o.o.p.PluginsService     ] [0eace0b2164c] loaded plugin [opensearch-ml]
2023-11-15T19:16:42.620Z INFO  [OpensearchProcessImpl] [2023-11-15T19:16:42,613][INFO ][o.o.p.PluginsService     ] [0eace0b2164c] loaded plugin [opensearch-observability]
2023-11-15T19:16:42.621Z INFO  [OpensearchProcessImpl] [2023-11-15T19:16:42,613][INFO ][o.o.p.PluginsService     ] [0eace0b2164c] loaded plugin [opensearch-security]
2023-11-15T19:16:42.633Z INFO  [OpensearchProcessImpl] [2023-11-15T19:16:42,633][INFO ][o.o.s.OpenSearchSecurityPlugin] [0eace0b2164c] Disabled https compression by default to mitigate BREACH attacks. You can enable it by setting 'http.compression: true' in opensearch.yml
2023-11-15T19:16:42.637Z INFO  [OpensearchProcessImpl] [2023-11-15T19:16:42,637][INFO ][o.o.e.ExtensionsManager  ] [0eace0b2164c] ExtensionsManager initialized
2023-11-15T19:16:42.662Z INFO  [OpensearchProcessImpl] [2023-11-15T19:16:42,661][INFO ][o.o.e.NodeEnvironment    ] [0eace0b2164c] using [1] data paths, mounts [[/var/lib/graylog-datanode (/dev/sda3)]], net usable_space [16.6gb], net total_space [34.8gb], types [ext4]
2023-11-15T19:16:42.662Z INFO  [OpensearchProcessImpl] [2023-11-15T19:16:42,662][INFO ][o.o.e.NodeEnvironment    ] [0eace0b2164c] heap size [1gb], compressed ordinary object pointers [true]
2023-11-15T19:16:42.749Z INFO  [OpensearchProcessImpl] [2023-11-15T19:16:42,749][INFO ][o.o.n.Node               ] [0eace0b2164c] node name [0eace0b2164c], node ID [SC2tH7MHTI2OruF6dRHXZg], cluster name [datanode-cluster], roles [ingest, remote_cluster_client, data, cluster_manager]
2023-11-15T19:16:45.261Z INFO  [OpensearchProcessImpl] [2023-11-15T19:16:45,260][WARN ][o.o.s.s.SecureSSLSettings] [0eace0b2164c] Setting [plugins.security.ssl.http.truststore_password] has a secure counterpart [plugins.security.ssl.http.truststore_password_secure] which should be used instead - allowing for legacy SSL setups
2023-11-15T19:16:45.262Z INFO  [OpensearchProcessImpl] [2023-11-15T19:16:45,262][WARN ][o.o.s.s.SecureSSLSettings] [0eace0b2164c] Setting [plugins.security.ssl.transport.truststore_password] has a secure counterpart [plugins.security.ssl.transport.truststore_password_secure] which should be used instead - allowing for legacy SSL setups
2023-11-15T19:16:45.266Z INFO  [OpensearchProcessImpl] [2023-11-15T19:16:45,266][WARN ][o.o.s.s.SecureSSLSettings] [0eace0b2164c] Setting [plugins.security.ssl.http.keystore_password] has a secure counterpart [plugins.security.ssl.http.keystore_password_secure] which should be used instead - allowing for legacy SSL setups
2023-11-15T19:16:45.267Z INFO  [OpensearchProcessImpl] [2023-11-15T19:16:45,266][WARN ][o.o.s.s.SecureSSLSettings] [0eace0b2164c] Setting [plugins.security.ssl.transport.keystore_password] has a secure counterpart [plugins.security.ssl.transport.keystore_password_secure] which should be used instead - allowing for legacy SSL setups
2023-11-15T19:16:46.631Z INFO  [OpensearchProcessImpl] [2023-11-15T19:16:46,630][WARN ][o.o.s.c.Salt             ] [0eace0b2164c] If you plan to use field masking pls configure compliance salt e1ukloTsQlOgPquJ to be a random string of 16 chars length identical on all nodes
2023-11-15T19:16:46.695Z INFO  [OpensearchProcessImpl] [2023-11-15T19:16:46,694][ERROR][o.o.s.a.s.SinkProvider   ] [0eace0b2164c] Default endpoint could not be created, auditlog will not work properly.
2023-11-15T19:16:46.697Z INFO  [OpensearchProcessImpl] [2023-11-15T19:16:46,697][WARN ][o.o.s.a.r.AuditMessageRouter] [0eace0b2164c] No default storage available, audit log may not work properly. Please check configuration.
2023-11-15T19:16:46.697Z INFO  [OpensearchProcessImpl] [2023-11-15T19:16:46,697][INFO ][o.o.s.a.i.AuditLogImpl   ] [0eace0b2164c] Message routing enabled: false
2023-11-15T19:16:46.762Z INFO  [OpensearchProcessImpl] [2023-11-15T19:16:46,762][INFO ][o.o.s.f.SecurityFilter   ] [0eace0b2164c] <NONE> indices are made immutable.
2023-11-15T19:16:47.058Z INFO  [OpensearchProcessImpl] [2023-11-15T19:16:47,052][INFO ][o.o.m.b.MLCircuitBreakerService] [0eace0b2164c] Registered ML memory breaker.
2023-11-15T19:16:47.059Z INFO  [OpensearchProcessImpl] [2023-11-15T19:16:47,053][INFO ][o.o.m.b.MLCircuitBreakerService] [0eace0b2164c] Registered ML disk breaker.
2023-11-15T19:16:47.059Z INFO  [OpensearchProcessImpl] [2023-11-15T19:16:47,053][INFO ][o.o.m.b.MLCircuitBreakerService] [0eace0b2164c] Registered ML native memory breaker.
2023-11-15T19:16:47.197Z INFO  [OpensearchProcessImpl] [2023-11-15T19:16:47,197][INFO ][o.r.Reflections          ] [0eace0b2164c] Reflections took 57 ms to scan 1 urls, producing 17 keys and 43 values 
2023-11-15T19:16:47.578Z INFO  [OpensearchProcessImpl] [2023-11-15T19:16:47,578][INFO ][o.o.a.b.ADCircuitBreakerService] [0eace0b2164c] Registered memory breaker.
2023-11-15T19:16:48.348Z INFO  [OpensearchProcessImpl] [2023-11-15T19:16:48,348][INFO ][o.o.t.NettyAllocator     ] [0eace0b2164c] creating NettyAllocator with the following configs: [name=unpooled, suggested_max_allocation_size=256kb, factors={opensearch.unsafe.use_unpooled_allocator=null, g1gc_enabled=true, g1gc_region_size=1mb, heap_size=1gb}]
2023-11-15T19:16:48.596Z INFO  [OpensearchProcessImpl] [2023-11-15T19:16:48,595][INFO ][o.o.d.DiscoveryModule    ] [0eace0b2164c] using discovery type [zen] and seed hosts providers [settings, file]
2023-11-15T19:16:49.079Z INFO  [OpensearchProcessImpl] [2023-11-15T19:16:49,078][WARN ][o.o.g.DanglingIndicesState] [0eace0b2164c] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
2023-11-15T19:16:49.375Z WARN  [OpensearchNodeHeartbeat] Opensearch REST api of process 1058 unavailable. Cause: Connection refused
2023-11-15T19:16:49.605Z INFO  [OpensearchProcessImpl] [2023-11-15T19:16:49,605][INFO ][o.o.n.Node               ] [0eace0b2164c] initialized
2023-11-15T19:16:49.606Z INFO  [OpensearchProcessImpl] [2023-11-15T19:16:49,605][INFO ][o.o.n.Node               ] [0eace0b2164c] starting ...
2023-11-15T19:16:49.742Z INFO  [OpensearchProcessImpl] [2023-11-15T19:16:49,742][INFO ][o.o.t.TransportService   ] [0eace0b2164c] publish_address {192.168.144.3:9300}, bound_addresses {0.0.0.0:9300}
2023-11-15T19:16:49.745Z INFO  [OpensearchProcessImpl] [2023-11-15T19:16:49,744][INFO ][o.o.t.TransportService   ] [0eace0b2164c] Remote clusters initialized successfully.
2023-11-15T19:16:49.915Z INFO  [OpensearchProcessImpl] [2023-11-15T19:16:49,915][INFO ][o.o.b.BootstrapChecks    ] [0eace0b2164c] bound or publishing to a non-loopback address, enforcing bootstrap checks
2023-11-15T19:16:49.927Z WARN  [OpensearchProcessImpl] ERROR: [1] bootstrap checks failed
2023-11-15T19:16:49.928Z WARN  [OpensearchProcessImpl] [1]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
2023-11-15T19:16:49.928Z WARN  [OpensearchProcessImpl] ERROR: OpenSearch did not exit normally - check the logs at /var/lib/graylog-datanode/opensearch/logs/8b2841e7-04e2-4ed2-9216-ed0e4e75f44b/datanode-cluster.log
2023-11-15T19:16:49.933Z INFO  [OpensearchProcessImpl] [2023-11-15T19:16:49,931][INFO ][o.o.n.Node               ] [0eace0b2164c] stopping ...
2023-11-15T19:16:49.935Z INFO  [OpensearchProcessImpl] [2023-11-15T19:16:49,934][INFO ][o.o.s.a.r.AuditMessageRouter] [0eace0b2164c] Closing AuditMessageRouter
2023-11-15T19:16:49.936Z INFO  [OpensearchProcessImpl] [2023-11-15T19:16:49,935][INFO ][o.o.s.a.s.SinkProvider   ] [0eace0b2164c] Closing DebugSink
2023-11-15T19:16:49.947Z INFO  [OpensearchProcessImpl] [2023-11-15T19:16:49,947][INFO ][o.o.n.Node               ] [0eace0b2164c] stopped
2023-11-15T19:16:49.948Z INFO  [OpensearchProcessImpl] [2023-11-15T19:16:49,947][INFO ][o.o.n.Node               ] [0eace0b2164c] closing ...
2023-11-15T19:16:49.963Z INFO  [OpensearchProcessImpl] [2023-11-15T19:16:49,963][INFO ][o.o.s.a.i.AuditLogImpl   ] [0eace0b2164c] Closing AuditLogImpl
2023-11-15T19:16:49.970Z INFO  [OpensearchProcessImpl] [2023-11-15T19:16:49,969][INFO ][o.o.n.Node               ] [0eace0b2164c] closed
2023-11-15T19:16:50.160Z WARN  [OpensearchProcessImpl] Opensearch process failed
org.apache.commons.exec.ExecuteException: Process exited with an error: 78 (Exit value: 78)
	at org.apache.commons.exec.DefaultExecutor.executeInternal(DefaultExecutor.java:404) ~[commons-exec-1.3.jar:1.3]
	at org.apache.commons.exec.DefaultExecutor.access$200(DefaultExecutor.java:48) ~[commons-exec-1.3.jar:1.3]
	at org.apache.commons.exec.DefaultExecutor$1.run(DefaultExecutor.java:200) [commons-exec-1.3.jar:1.3]
	at java.lang.Thread.run(Unknown Source) [?:?]
2023-11-15T19:16:50.162Z WARN  [ProcessWatchdog] Process watchdog terminated after too many restart attempts
@zikkuratti
Copy link

YEAH ME TO CANT START

@janheise
Copy link
Contributor

Hi @danielescalona @zikkuratti can you both please provide a little more context? You both successfully ran the Preflight/Configuration and OpenSearch is not coming up after pressing the resume button?

Is this both on Docker under Linux? Did you set the sysctl -w vm.max_map_count=262144 setting, which is necessary for OpenSearch (maybe this should be more prominent in our docs...)?

Do you just run the defaults? In that case, the easiest way to start over is probably a docker compose down -v and start again.

@janheise janheise self-assigned this Nov 16, 2023
@danielescalona
Copy link
Author

I was able to get the login and create the CA. Once I hit next to create the CA, it just hangs and that is the error I get. I will run the sysctl -w vm.max_map_count=262144 and get back to you. That thing is not on the docs. All default with the exception of the password. I am using the latest docker and alpine linux

@janheise
Copy link
Contributor

janheise commented Nov 16, 2023 •
edited
Loading

@danielescalona you're correct, it did not end up in the docs, so here is some more info. I'll make sure that it get's fixed. I hope you get it to work now. Please make sure that you make the setting permanent, see https://opensearch.org/docs/latest/install-and-configure/install-opensearch/index/#important-settings

@zikkuratti
Copy link

it try to connect elastic search did you just docker compose up file inside open-core?

@janheise
Copy link
Contributor

@zikkuratti can you please give more info? it's OpenSearch in the docker-compose.yml we're talking about. I saw that you forked/edited a docker-compose.yml that references Elasticsearch. Are we talking about the problem from the top of the page or do you have a different problem? I just tested the file from the open-core subdir and it worked fine here right out of the box.

@zikkuratti
Copy link

no i cant no host on this pc , no fork i just compile your default and its stuck

@janheise
Copy link
Contributor

then please post logs

@zikkuratti
Copy link

i just need info your system start out of the box like it is?

@janheise
Copy link
Contributor

janheise commented Nov 17, 2023 •
edited
Loading

If you want my help, please answer my questions, too. As I wrote: it works without any modification in the docker-compose.yml- you have to create the .env with your credentials and set the sysctl -w vm.max_map_count=262144 prior to starting on your docker host. But that's it. If it's not working for you, please post exact details of your setup and the log output. Maybe even list the exact steps what you're doing.

@janheise janheise changed the title datanode DataNode is not starting up after provisioning certificates Nov 17, 2023
@danielescalona
Copy link
Author

Hi @danielescalona @zikkuratti can you both please provide a little more context? You both successfully ran the Preflight/Configuration and OpenSearch is not coming up after pressing the resume button?

Is this both on Docker under Linux? Did you set the sysctl -w vm.max_map_count=262144 setting, which is necessary for OpenSearch (maybe this should be more prominent in our docs...)?

Do you just run the defaults? In that case, the easiest way to start over is probably a docker compose down -v and start again.

I can run it with the sysctl modification. Thanks!

@janheise
Copy link
Contributor

@zikkuratti were you successful by now?

@janheise
Copy link
Contributor

I'm closing this as it seems that the problems are solved

@danielescalona
Copy link
Author

danielescalona commented Nov 21, 2023 via email

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@janheise janheise

Labels
triaged
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@kroepke @janheise @zikkuratti @danielescalona