Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Install never completes in log - hangs on Kibana #541

Open
SecOpsWarrior opened this issue Jan 28, 2021 · 8 comments
Open

Install never completes in log - hangs on Kibana #541

SecOpsWarrior opened this issue Jan 28, 2021 · 8 comments

Comments

@SecOpsWarrior
Copy link

SecOpsWarrior commented Jan 28, 2021 •
edited

Describe the problem

I've tried running clean installs 3-4 times with the same issue on ubuntu 20.04.2 LTS (Focal Fossa)

  • Tried installing versions 2, and 4 of script (output below is from version 4) - same issue
  • Tried uninstalling, wiping and reinstalling docker prior to installing - same issue

Installation hangs on [HELK-INSTALLATION-INFO] Waiting for some services to be up .....

And in the install logs, it seems when it gets to kibana, it never gets to "done"

Pulling helk-spark-master (otrf/helk-spark-master:2.4.5)...
Creating helk-elasticsearch ... done
Creating helk-kibana ... done
Status: Downloaded newer image for otrf/helk-spark-master:2.4.5
Creating helk-logstash ... done
2.4.5: Pulling from otrf/helk-spark-worker
Digest: sha256:0c3e2f759d6f286dbf740dab6a74740eb1b173d41156d50c3e4a32ea7e5aa74c
Status: Downloaded newer image for otrf/helk-spark-worker:2.4.5
Creating helk-zookeeper ... done
Creating helk-spark-master ... done
Creating helk-spark-worker ... done
Creating helk-ksql-server ... done
Creating helk-ksql-cli ... done
Creating helk-kibana ...
Creating helk-nginx ...
Creating helk-logstash ...
Creating helk-spark-master ...
Creating helk-jupyter ...
Creating helk-zookeeper ...
Creating helk-elastalert ...
Creating helk-kafka-broker ...
Creating helk-spark-worker ...
Creating helk-ksql-server ...
Creating helk-ksql-cli ...

Provide the output of the following commands

Get operating system and version

NAME="Ubuntu"
VERSION="20.04.2 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.2 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal

Get disk space, memory, processor cores, and docker storage

Docker Space:
Filesystem Size Used Avail Use% Mounted on
/dev/sda2 1007G 286G 671G 30% /

Memory:
total used free shared buff/cache available
Mem: 15 13 0 0 2 1
Swap: 3 3 0

Cores:
8

Get output of the HELK docker containers:

NOTE:

  • If I wait a while, it will say Its time for hunting, but logging in doesnt work.
  • I get "Kibana server is not ready yet" when browsing to web interface

Docker shows it all running

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
995ec3c0b0f2 confluentinc/cp-ksql-cli:5.1.3 "/bin/sh" 5 minutes ago Up 5 minutes helk-ksql-cli
1139550ed979 confluentinc/cp-ksql-server:5.1.3 "/etc/confluent/dock…" 5 minutes ago Up 5 minutes 0.0.0.0:8088->8088/tcp helk-ksql-server
e0711a5c58b5 otrf/helk-spark-worker:2.4.5 "./spark-worker-entr…" 5 minutes ago Up 5 minutes helk-spark-worker
ae754782be12 otrf/helk-kafka-broker:2.4.0 "./kafka-entrypoint.…" 5 minutes ago Up 5 minutes 0.0.0.0:9092->9092/tcp helk-kafka-broker
e6a2f5106a95 otrf/helk-elastalert:latest "./elastalert-entryp…" 5 minutes ago Up 5 minutes helk-elastalert
a839e6ab032a otrf/helk-zookeeper:2.4.0 "./zookeeper-entrypo…" 5 minutes ago Up 5 minutes 2181/tcp, 2888/tcp, 3888/tcp helk-zookeeper
03c3623488ef otrf/helk-spark-master:2.4.5 "./spark-master-entr…" 5 minutes ago Up 5 minutes 7077/tcp, 0.0.0.0:8080->8080/tcp helk-spark-master
6d78bd249a7e docker_helk-jupyter "/opt/jupyter/script…" 5 minutes ago Up 5 minutes 8000/tcp, 8888/tcp helk-jupyter
4746ea18d3b9 otrf/helk-logstash:7.6.2.1 "/usr/share/logstash…" 5 minutes ago Up 5 minutes 0.0.0.0:3515->3515/tcp, 0.0.0.0:5044->5044/tcp, 0.0.0.0:5514->5514/tcp, 0.0.0.0:5514->5514/udp, 0.0.0.0:8515-8516->8515-8516/tcp, 0.0.0.0:8531->8531/tcp, 0.0.0.0:8515-8516->8515-8516/udp, 9600/tcp helk-logstash
6311e1a5131d otrf/helk-nginx:0.3.0 "/opt/helk/scripts/n…" 5 minutes ago Up 5 minutes 0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp helk-nginx
9f586664f876 docker.elastic.co/kibana/kibana:7.6.2 "/usr/share/kibana/s…" 5 minutes ago Up 5 minutes 5601/tcp helk-kibana
fdcf51e9bf24 docker.elastic.co/elasticsearch/elasticsearch:7.6.2 "/usr/share/elastics…" 5 minutes ago Up 5 minutes 9200/tcp, 9300/tcp helk-elasticsearch

Docker logs for kibana show the following:

HELK-KIBANA-DOCKER-ENTRYPOINT-INFO: Setting Elasticsearch URL to http://helk-elasticsearch:9200
HELK-KIBANA-DOCKER-ENTRYPOINT-INFO: Setting Kibana server to helk-kibana
HELK-KIBANA-DOCKER-ENTRYPOINT-INFO: Setting Kibana server port to 5601
HELK-KIBANA-DOCKER-ENTRYPOINT-INFO: Setting Kibana URL to http://helk-kibana:5601
Enter host password for user '':
HELK-KIBANA-DOCKER-ENTRYPOINT-INFO: Waiting for very basic elasticsearch check..
Enter host password for user '':
HELK-KIBANA-DOCKER-ENTRYPOINT-INFO: Waiting for very basic elasticsearch check..
Enter host password for user '':
HELK-KIBANA-DOCKER-ENTRYPOINT-INFO: Waiting for very basic elasticsearch check..
Enter host password for user '':
HELK-KIBANA-DOCKER-ENTRYPOINT-INFO: Waiting for very basic elasticsearch check..
Enter host password for user '':
HELK-KIBANA-DOCKER-ENTRYPOINT-INFO: Waiting for very basic elasticsearch check..
Enter host password for user '':
HELK-KIBANA-DOCKER-ENTRYPOINT-INFO: Waiting for very basic elasticsearch check..
Enter host password for user '':
HELK-KIBANA-DOCKER-ENTRYPOINT-INFO: Waiting for very basic elasticsearch check..
Enter host password for user '':
HELK-KIBANA-DOCKER-ENTRYPOINT-INFO: Waiting for very basic elasticsearch check..
Enter host password for user '':
HELK-KIBANA-DOCKER-ENTRYPOINT-INFO: Waiting for very basic elasticsearch check..
Enter host password for user '':
HELK-KIBANA-DOCKER-ENTRYPOINT-INFO: Waiting for very basic elasticsearch check..
Enter host password for user '':
HELK-KIBANA-DOCKER-ENTRYPOINT-INFO: Waiting for very basic elasticsearch check..
Enter host password for user '':
HELK-KIBANA-DOCKER-ENTRYPOINT-INFO: Waiting for very basic elasticsearch check..
Enter host password for user '':
HELK-KIBANA-DOCKER-ENTRYPOINT-INFO: Waiting for very basic elasticsearch check..
Enter host password for user '':
HELK-KIBANA-DOCKER-ENTRYPOINT-INFO: Waiting for very basic elasticsearch check..
Enter host password for user '':
HELK-KIBANA-DOCKER-ENTRYPOINT-INFO: Starting Kibana service..
HELK-KIBANA-DOCKER-ENTRYPOINT-INFO: Running kibana-setup.sh..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for Kibana internal port to be up..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for Kibana internal port to be up..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for Kibana internal port to be up..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for Kibana internal port to be up..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for Kibana internal port to be up..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for Kibana internal port to be up..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for Kibana internal port to be up..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for Kibana internal port to be up..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for Kibana internal port to be up..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for Kibana internal port to be up..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for Kibana internal port to be up..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for Kibana internal port to be up..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for Kibana internal port to be up..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for Kibana internal port to be up..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for Kibana internal port to be up..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for Kibana internal port to be up..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for Kibana internal port to be up..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for Kibana internal port to be up..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for Kibana internal port to be up..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for Kibana internal port to be up..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for Kibana internal port to be up..
HELK-KIBANA-DOCKER-SETUP-INFO: Kibana internal port is up..
HELK-KIBANA-DOCKER-SETUP-INFO: Checking elasticsearch '.kibana' index
Enter host password for user '':
HELK-KIBANA-DOCKER-SETUP-INFO: Elasticsearch '.kibana' index is up..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for kibana server..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for kibana server..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for kibana server..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for kibana server..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for kibana server..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for kibana server..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for kibana server..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for kibana server..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for kibana server..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for kibana server..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for kibana server..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for kibana server..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for kibana server..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for kibana server..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for kibana server..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for kibana server..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for kibana server..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for kibana server..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for kibana server..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for kibana server..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for kibana server..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for kibana server..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for kibana server..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for kibana server..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for kibana server..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for kibana server..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for kibana server..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for kibana server..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for kibana server..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for kibana server..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for kibana server..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for kibana server..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for kibana server..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for kibana server..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for kibana server..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for kibana server..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for kibana server..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for kibana server..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for kibana server..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for kibana server..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for kibana server..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for kibana server..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for kibana server..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for kibana server..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for kibana server..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for kibana server..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for kibana server..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for kibana server..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for kibana server..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for kibana server..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for kibana server..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for kibana server..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for kibana server..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for kibana server..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for kibana server..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for kibana server..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for kibana server..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for kibana server..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for kibana server..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for kibana server..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for kibana server..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for kibana server..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for kibana server..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for kibana server..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for kibana server..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for kibana server..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for kibana server..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for kibana server..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for kibana server..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for kibana server..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for kibana server..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for kibana server..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for kibana server..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for kibana server..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for kibana server..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for kibana server..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for kibana server..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for kibana server..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for kibana server..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for kibana server..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for kibana server..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for kibana server..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for kibana server..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for kibana server..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for kibana server..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for kibana server..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for kibana server..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for kibana server..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for kibana server..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for kibana server..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for kibana server..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for kibana server..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for kibana server..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for kibana server..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for kibana server..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for kibana server..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for kibana server..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for kibana server..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for kibana server..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for kibana server..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for kibana server..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for kibana server..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for kibana server..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for kibana server..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for kibana server..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for kibana server..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for kibana server..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for kibana server..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for kibana server..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for kibana server..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for kibana server..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for kibana server..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for kibana server..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for kibana server..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for kibana server..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for kibana server..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for kibana server..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for kibana server..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for kibana server..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for kibana server..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for kibana server..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for kibana server..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for kibana server..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for kibana server..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for kibana server..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for kibana server..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for kibana server..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for kibana server..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for kibana server..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for kibana server..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for kibana server..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for kibana server..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for kibana server..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for kibana server..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for kibana server..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for kibana server..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for kibana server..

I am stumped...any ideas?

I am running the install script using sudo in the home directory of the user.
@SecOpsWarrior SecOpsWarrior changed the title nginx 502 error - also, install never completes in log Install never completes in log - hangs on Kibana Jan 28, 2021
@SecOpsWarrior
Copy link
Author

SecOpsWarrior commented Jan 29, 2021 •
edited

UPDATE: after a really long time, now the error

502 Bad Gateway
nginx

pops up when trying to access kibana

I went back through kibana logs and saw this...

HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for kibana server..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for kibana server..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for kibana server..

FATAL [circuit_breaking_exception] [parent] Data too large, data for [<http_request>] would be [6410688696/5.9gb], which is larger than the limit of [6408713011/5.9gb], real usage: [6410686416/5.9gb], new bytes reserved: [2280/2.2kb], usages [request=65536/64kb, fielddata=47709212/45.4mb, in_flight_requests=13084/12.7kb, accounting=555321383/529.5mb], with { bytes_wanted=6410688696 & bytes_limit=6408713011 & durability="PERMANENT" } :: {"path":"/.kibana/_count","query":{},"body":"{"query":{"bool":{"should":[{"bool":{"must":[{"exists":{"field":"graph-workspace"}},{"bool":{"must_not":{"term":{"migrationVersion.graph-workspace":"7.0.0"}}}}]}},{"bool":{"must":[{"exists":{"field":"space"}},{"bool":{"must_not":{"term":{"migrationVersion.space":"6.6.0"}}}}]}},{"bool":{"must":[{"exists":{"field":"map"}},{"bool":{"must_not":{"term":{"migrationVersion.map":"7.6.0"}}}}]}},{"bool":{"must":[{"exists":{"field":"canvas-workpad"}},{"bool":{"must_not":{"term":{"migrationVersion.canvas-workpad":"7.0.0"}}}}]}},{"bool":{"must":[{"exists":{"field":"task"}},{"bool":{"must_not":{"term":{"migrationVersion.task":"7.6.0"}}}}]}},{"bool":{"must":[{"exists":{"field":"index-pattern"}},{"bool":{"must_not":{"term":{"migrationVersion.index-pattern":"7.6.0"}}}}]}},{"bool":{"must":[{"exists":{"field":"visualization"}},{"bool":{"must_not":{"term":{"migrationVersion.visualization":"7.4.2"}}}}]}},{"bool":{"must":[{"exists":{"field":"dashboard"}},{"bool":{"must_not":{"term":{"migrationVersion.dashboard":"7.3.0"}}}}]}},{"bool":{"must":[{"exists":{"field":"search"}},{"bool":{"must_not":{"term":{"migrationVersion.search":"7.4.0"}}}}]}}]}}}","statusCode":429,"response":"{"error":{"root_cause":[{"type":"circuit_breaking_exception","reason":"[parent] Data too large, data for [<http_request>] would be [6410688696/5.9gb], which is larger than the limit of [6408713011/5.9gb], real usage: [6410686416/5.9gb], new bytes reserved: [2280/2.2kb], usages [request=65536/64kb, fielddata=47709212/45.4mb, in_flight_requests=13084/12.7kb, accounting=555321383/529.5mb]","bytes_wanted":6410688696,"bytes_limit":6408713011,"durability":"PERMANENT"}],"type":"circuit_breaking_exception","reason":"[parent] Data too large, data for [<http_request>] would be [6410688696/5.9gb], which is larger than the limit of [6408713011/5.9gb], real usage: [6410686416/5.9gb], new bytes reserved: [2280/2.2kb], usages [request=65536/64kb, fielddata=47709212/45.4mb, in_flight_requests=13084/12.7kb, accounting=555321383/529.5mb]","bytes_wanted":6410688696,"bytes_limit":6408713011,"durability":"PERMANENT"},"status":429}"}

HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for kibana server..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for kibana server..
HELK-KIBANA-DOCKER-SETUP-INFO: Waiting for kibana server..

I looked at the /docker/helk-elasticsearch/config/jvm.options config file and it says:

# Xms represents the initial size of total heap space
# Xmx represents the maximum size of total heap space

# This will get set by HELK scripts
#-Xms1g
#-Xmx1g

Is there a way to modify the scripts to increase this to 10g? and if so, where is the script I would modify?

OR

Is this a memory error in Kibana somewhere that's consuming all available memory until it reaches the circuit breaker and causes it to terminate?

@priamai
Copy link

priamai commented Feb 9, 2021

Yes that is right you should increase your RAM available to Docker, I can see from your logs that you are maxed out and it using already the file swap.
Can you try install version 1 (although not a big difference)?
Can you also try to install a simple vanilla ELK docker stack and see if you can manage that in memory?
I think quicker solution will be to buy some RAM .

@SecOpsWarrior
Copy link
Author

SecOpsWarrior commented Feb 9, 2021 via email •
edited

@SecOpsWarrior
Copy link
Author

Got around to messing with this a bit more. Completely wiped the server, reinstalled docker, and then attempted install. I looks like the kibana server is actually waiting on elasticsearch to start, but elasticsearch is the one having the issue.

What could be causing this? This is a brand new install.

sudo docker logs --follow --tail 100 helk-elasticsearch
"at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:125) ~[elasticsearch-cli-7.6.2.jar:7.6.2]",
"at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-7.6.2.jar:7.6.2]",
"at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:126) ~[elasticsearch-7.6.2.jar:7.6.2]",
"at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-7.6.2.jar:7.6.2]",
"Caused by: org.elasticsearch.ElasticsearchException: failed to bind service",
"at org.elasticsearch.node.Node.(Node.java:615) ~[elasticsearch-7.6.2.jar:7.6.2]",
"at org.elasticsearch.node.Node.(Node.java:257) ~[elasticsearch-7.6.2.jar:7.6.2]",
"at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:221) ~[elasticsearch-7.6.2.jar:7.6.2]",
"at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:221) ~[elasticsearch-7.6.2.jar:7.6.2]",
"at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:349) ~[elasticsearch-7.6.2.jar:7.6.2]",
"at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) ~[elasticsearch-7.6.2.jar:7.6.2]",
"... 6 more",
"Caused by: org.apache.lucene.index.CorruptIndexException: Unexpected file read error while reading index. (resource=BufferedChecksumIndexInput(SimpleFSIndexInput(path="/usr/share/elasticsearch/data/nodes/0/_state/segments_7ca0")))",
"at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:291) ~[lucene-core-8.4.0.jar:8.4.0 bc02ab906445fcf4e297f4ef00ab4a54fdd72ca2 - jpountz - 2019-12-19 20:16:14]",
"at org.apache.lucene.index.StandardDirectoryReader$1.doBody(StandardDirectoryReader.java:64) ~[lucene-core-8.4.0.jar:8.4.0 bc02ab906445fcf4e297f4ef00ab4a54fdd72ca2 - jpountz - 2019-12-19 20:16:14]",
"at org.apache.lucene.index.StandardDirectoryReader$1.doBody(StandardDirectoryReader.java:61) ~[lucene-core-8.4.0.jar:8.4.0 bc02ab906445fcf4e297f4ef00ab4a54fdd72ca2 - jpountz - 2019-12-19 20:16:14]",
"at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:680) ~[lucene-core-8.4.0.jar:8.4.0 bc02ab906445fcf4e297f4ef00ab4a54fdd72ca2 - jpountz - 2019-12-19 20:16:14]",
"at org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:84) ~[lucene-core-8.4.0.jar:8.4.0 bc02ab906445fcf4e297f4ef00ab4a54fdd72ca2 - jpountz - 2019-12-19 20:16:14]",
"at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:76) ~[lucene-core-8.4.0.jar:8.4.0 bc02ab906445fcf4e297f4ef00ab4a54fdd72ca2 - jpountz - 2019-12-19 20:16:14]",
"at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:64) ~[lucene-core-8.4.0.jar:8.4.0 bc02ab906445fcf4e297f4ef00ab4a54fdd72ca2 - jpountz - 2019-12-19 20:16:14]",
"at org.elasticsearch.gateway.PersistedClusterStateService.nodeMetaData(PersistedClusterStateService.java:262) ~[elasticsearch-7.6.2.jar:7.6.2]",
"at org.elasticsearch.env.NodeEnvironment.loadNodeMetaData(NodeEnvironment.java:408) ~[elasticsearch-7.6.2.jar:7.6.2]",
"at org.elasticsearch.env.NodeEnvironment.(NodeEnvironment.java:329) ~[elasticsearch-7.6.2.jar:7.6.2]",
"at org.elasticsearch.node.Node.(Node.java:277) ~[elasticsearch-7.6.2.jar:7.6.2]",
"at org.elasticsearch.node.Node.(Node.java:257) ~[elasticsearch-7.6.2.jar:7.6.2]",
uncaught exception in thread [main]
"at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:221) ~[elasticsearch-7.6.2.jar:7.6.2]",
"at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:221) ~[elasticsearch-7.6.2.jar:7.6.2]",
"at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:349) ~[elasticsearch-7.6.2.jar:7.6.2]",
"at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) ~[elasticsearch-7.6.2.jar:7.6.2]",
"... 6 more",
"Caused by: java.nio.file.NoSuchFileException: /usr/share/elasticsearch/data/nodes/0/_state/_78qd.si",
"at sun.nio.fs.UnixException.translateToIOException(UnixException.java:92) ~[?:?]",
"at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) ~[?:?]",
"at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116) ~[?:?]",
"at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:219) ~[?:?]",
"at java.nio.file.Files.newByteChannel(Files.java:374) ~[?:?]",
"at java.nio.file.Files.newByteChannel(Files.java:425) ~[?:?]",
"at org.apache.lucene.store.SimpleFSDirectory.openInput(SimpleFSDirectory.java:77) ~[lucene-core-8.4.0.jar:8.4.0 bc02ab906445fcf4e297f4ef00ab4a54fdd72ca2 - jpountz - 2019-12-19 20:16:14]",
"at org.apache.lucene.store.Directory.openChecksumInput(Directory.java:157) ~[lucene-core-8.4.0.jar:8.4.0 bc02ab906445fcf4e297f4ef00ab4a54fdd72ca2 - jpountz - 2019-12-19 20:16:14]",
"at org.apache.lucene.codecs.lucene70.Lucene70SegmentInfoFormat.read(Lucene70SegmentInfoFormat.java:91) ~[lucene-core-8.4.0.jar:8.4.0 bc02ab906445fcf4e297f4ef00ab4a54fdd72ca2 - jpountz - 2019-12-19 20:16:14]",
"at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:353) ~[lucene-core-8.4.0.jar:8.4.0 bc02ab906445fcf4e297f4ef00ab4a54fdd72ca2 - jpountz - 2019-12-19 20:16:14]",
"at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:289) ~[lucene-core-8.4.0.jar:8.4.0 bc02ab906445fcf4e297f4ef00ab4a54fdd72ca2 - jpountz - 2019-12-19 20:16:14]",
"at org.apache.lucene.index.StandardDirectoryReader$1.doBody(StandardDirectoryReader.java:64) ~[lucene-core-8.4.0.jar:8.4.0 bc02ab906445fcf4e297f4ef00ab4a54fdd72ca2 - jpountz - 2019-12-19 20:16:14]",
"at org.apache.lucene.index.StandardDirectoryReader$1.doBody(StandardDirectoryReader.java:61) ~[lucene-core-8.4.0.jar:8.4.0 bc02ab906445fcf4e297f4ef00ab4a54fdd72ca2 - jpountz - 2019-12-19 20:16:14]",
"at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:680) ~[lucene-core-8.4.0.jar:8.4.0 bc02ab906445fcf4e297f4ef00ab4a54fdd72ca2 - jpountz - 2019-12-19 20:16:14]",
"at org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:84) ~[lucene-core-8.4.0.jar:8.4.0 bc02ab906445fcf4e297f4ef00ab4a54fdd72ca2 - jpountz - 2019-12-19 20:16:14]",
"at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:76) ~[lucene-core-8.4.0.jar:8.4.0 bc02ab906445fcf4e297f4ef00ab4a54fdd72ca2 - jpountz - 2019-12-19 20:16:14]",
"at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:64) ~[lucene-core-8.4.0.jar:8.4.0 bc02ab906445fcf4e297f4ef00ab4a54fdd72ca2 - jpountz - 2019-12-19 20:16:14]",
"at org.elasticsearch.gateway.PersistedClusterStateService.nodeMetaData(PersistedClusterStateService.java:262) ~[elasticsearch-7.6.2.jar:7.6.2]",
"at org.elasticsearch.env.NodeEnvironment.loadNodeMetaData(NodeEnvironment.java:408) ~[elasticsearch-7.6.2.jar:7.6.2]",
"at org.elasticsearch.env.NodeEnvironment.(NodeEnvironment.java:329) ~[elasticsearch-7.6.2.jar:7.6.2]",
"at org.elasticsearch.node.Node.(Node.java:277) ~[elasticsearch-7.6.2.jar:7.6.2]",
"at org.elasticsearch.node.Node.(Node.java:257) ~[elasticsearch-7.6.2.jar:7.6.2]",
"at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:221) ~[elasticsearch-7.6.2.jar:7.6.2]",
"at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:221) ~[elasticsearch-7.6.2.jar:7.6.2]",
"at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:349) ~[elasticsearch-7.6.2.jar:7.6.2]",
"at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) ~[elasticsearch-7.6.2.jar:7.6.2]",
"... 6 more"] }
ElasticsearchException[failed to bind service]; nested: CorruptIndexException[Unexpected file read error while reading index. (resource=BufferedChecksumIndexInput(SimpleFSIndexInput(path="/usr/share/elasticsearch/data/nodes/0/_state/segments_7ca0")))]; nested: NoSuchFileException[/usr/share/elasticsearch/data/nodes/0/_state/_78qd.si];
Likely root cause: java.nio.file.NoSuchFileException: /usr/share/elasticsearch/data/nodes/0/_state/_78qd.si
at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)
at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116)
at java.base/sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:219)
at java.base/java.nio.file.Files.newByteChannel(Files.java:374)
at java.base/java.nio.file.Files.newByteChannel(Files.java:425)
at org.apache.lucene.store.SimpleFSDirectory.openInput(SimpleFSDirectory.java:77)
at org.apache.lucene.store.Directory.openChecksumInput(Directory.java:157)
at org.apache.lucene.codecs.lucene70.Lucene70SegmentInfoFormat.read(Lucene70SegmentInfoFormat.java:91)
at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:353)
at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:289)
at org.apache.lucene.index.StandardDirectoryReader$1.doBody(StandardDirectoryReader.java:64)
at org.apache.lucene.index.StandardDirectoryReader$1.doBody(StandardDirectoryReader.java:61)
at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:680)
at org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:84)
at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:76)
at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:64)
at org.elasticsearch.gateway.PersistedClusterStateService.nodeMetaData(PersistedClusterStateService.java:262)
at org.elasticsearch.env.NodeEnvironment.loadNodeMetaData(NodeEnvironment.java:408)
at org.elasticsearch.env.NodeEnvironment.(NodeEnvironment.java:329)
at org.elasticsearch.node.Node.(Node.java:277)
at org.elasticsearch.node.Node.(Node.java:257)
at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:221)
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:221)
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:349)
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170)
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:161)
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86)
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:125)
at org.elasticsearch.cli.Command.main(Command.java:90)
<<>>
For complete error details, refer to the log at /usr/share/elasticsearch/logs/helk-cluster.log
#-Xms1g
#-Xmx1g
[HELK-ES-DOCKER-INSTALLATION-INFO] Setting ES_JAVA_OPTS to -Xms6500m -Xmx6500m from custom HELK "algorithm"
[HELK-ES-DOCKER-INSTALLATION-INFO] Running docker-entrypoint script..
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
{"type": "server", "timestamp": "2021-04-08T17:42:27,560Z", "level": "INFO", "component": "o.e.e.NodeEnvironment", "cluster.name": "helk-cluster", "node.name": "helk-1", "message": "using [1] data paths, mounts [[/usr/share/elasticsearch/data (/dev/sda2)]], net usable_space [663.3gb], net total_space [1006.9gb], types [ext4]" }
{"type": "server", "timestamp": "2021-04-08T17:42:27,565Z", "level": "INFO", "component": "o.e.e.NodeEnvironment", "cluster.name": "helk-cluster", "node.name": "helk-1", "message": "heap size [6.2gb], compressed ordinary object pointers [true]" }
{"type": "server", "timestamp": "2021-04-08T17:42:30,606Z", "level": "ERROR", "component": "o.e.b.ElasticsearchUncaughtExceptionHandler", "cluster.name": "helk-cluster", "node.name": "helk-1", "message": "uncaught exception in thread [main]",
"stacktrace": ["org.elasticsearch.bootstrap.StartupException: ElasticsearchException[failed to bind service]; nested: CorruptIndexException[Unexpected file read error while reading index. (resource=BufferedChecksumIndexInput(SimpleFSIndexInput(path="/usr/share/elasticsearch/data/nodes/0/_state/segments_7ca0")))]; nested: NoSuchFileException[/usr/share/elasticsearch/data/nodes/0/_state/_78qd.si];",
"at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:174) ~[elasticsearch-7.6.2.jar:7.6.2]",
"at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:161) ~[elasticsearch-7.6.2.jar:7.6.2]",
"at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-7.6.2.jar:7.6.2]",
"at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:125) ~[elasticsearch-cli-7.6.2.jar:7.6.2]",
"at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-7.6.2.jar:7.6.2]",
"at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:126) ~[elasticsearch-7.6.2.jar:7.6.2]",
"at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-7.6.2.jar:7.6.2]",
"Caused by: org.elasticsearch.ElasticsearchException: failed to bind service",
"at org.elasticsearch.node.Node.(Node.java:615) ~[elasticsearch-7.6.2.jar:7.6.2]",
"at org.elasticsearch.node.Node.(Node.java:257) ~[elasticsearch-7.6.2.jar:7.6.2]",
"at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:221) ~[elasticsearch-7.6.2.jar:7.6.2]",
"at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:221) ~[elasticsearch-7.6.2.jar:7.6.2]",
"at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:349) ~[elasticsearch-7.6.2.jar:7.6.2]",
"at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) ~[elasticsearch-7.6.2.jar:7.6.2]",
uncaught exception in thread [main]
"... 6 more",
"Caused by: org.apache.lucene.index.CorruptIndexException: Unexpected file read error while reading index. (resource=BufferedChecksumIndexInput(SimpleFSIndexInput(path="/usr/share/elasticsearch/data/nodes/0/_state/segments_7ca0")))",
"at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:291) ~[lucene-core-8.4.0.jar:8.4.0 bc02ab906445fcf4e297f4ef00ab4a54fdd72ca2 - jpountz - 2019-12-19 20:16:14]",
"at org.apache.lucene.index.StandardDirectoryReader$1.doBody(StandardDirectoryReader.java:64) ~[lucene-core-8.4.0.jar:8.4.0 bc02ab906445fcf4e297f4ef00ab4a54fdd72ca2 - jpountz - 2019-12-19 20:16:14]",
"at org.apache.lucene.index.StandardDirectoryReader$1.doBody(StandardDirectoryReader.java:61) ~[lucene-core-8.4.0.jar:8.4.0 bc02ab906445fcf4e297f4ef00ab4a54fdd72ca2 - jpountz - 2019-12-19 20:16:14]",
"at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:680) ~[lucene-core-8.4.0.jar:8.4.0 bc02ab906445fcf4e297f4ef00ab4a54fdd72ca2 - jpountz - 2019-12-19 20:16:14]",
"at org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:84) ~[lucene-core-8.4.0.jar:8.4.0 bc02ab906445fcf4e297f4ef00ab4a54fdd72ca2 - jpountz - 2019-12-19 20:16:14]",
"at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:76) ~[lucene-core-8.4.0.jar:8.4.0 bc02ab906445fcf4e297f4ef00ab4a54fdd72ca2 - jpountz - 2019-12-19 20:16:14]",
"at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:64) ~[lucene-core-8.4.0.jar:8.4.0 bc02ab906445fcf4e297f4ef00ab4a54fdd72ca2 - jpountz - 2019-12-19 20:16:14]",
"at org.elasticsearch.gateway.PersistedClusterStateService.nodeMetaData(PersistedClusterStateService.java:262) ~[elasticsearch-7.6.2.jar:7.6.2]",
"at org.elasticsearch.env.NodeEnvironment.loadNodeMetaData(NodeEnvironment.java:408) ~[elasticsearch-7.6.2.jar:7.6.2]",
"at org.elasticsearch.env.NodeEnvironment.(NodeEnvironment.java:329) ~[elasticsearch-7.6.2.jar:7.6.2]",
"at org.elasticsearch.node.Node.(Node.java:277) ~[elasticsearch-7.6.2.jar:7.6.2]",
"at org.elasticsearch.node.Node.(Node.java:257) ~[elasticsearch-7.6.2.jar:7.6.2]",
"at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:221) ~[elasticsearch-7.6.2.jar:7.6.2]",
"at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:221) ~[elasticsearch-7.6.2.jar:7.6.2]",
"at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:349) ~[elasticsearch-7.6.2.jar:7.6.2]",
"at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) ~[elasticsearch-7.6.2.jar:7.6.2]",
"... 6 more",
"Caused by: java.nio.file.NoSuchFileException: /usr/share/elasticsearch/data/nodes/0/_state/_78qd.si",
"at sun.nio.fs.UnixException.translateToIOException(UnixException.java:92) ~[?:?]",
"at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) ~[?:?]",
"at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116) ~[?:?]",
"at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:219) ~[?:?]",
"at java.nio.file.Files.newByteChannel(Files.java:374) ~[?:?]",
"at java.nio.file.Files.newByteChannel(Files.java:425) ~[?:?]",
"at org.apache.lucene.store.SimpleFSDirectory.openInput(SimpleFSDirectory.java:77) ~[lucene-core-8.4.0.jar:8.4.0 bc02ab906445fcf4e297f4ef00ab4a54fdd72ca2 - jpountz - 2019-12-19 20:16:14]",
"at org.apache.lucene.store.Directory.openChecksumInput(Directory.java:157) ~[lucene-core-8.4.0.jar:8.4.0 bc02ab906445fcf4e297f4ef00ab4a54fdd72ca2 - jpountz - 2019-12-19 20:16:14]",
"at org.apache.lucene.codecs.lucene70.Lucene70SegmentInfoFormat.read(Lucene70SegmentInfoFormat.java:91) ~[lucene-core-8.4.0.jar:8.4.0 bc02ab906445fcf4e297f4ef00ab4a54fdd72ca2 - jpountz - 2019-12-19 20:16:14]",
"at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:353) ~[lucene-core-8.4.0.jar:8.4.0 bc02ab906445fcf4e297f4ef00ab4a54fdd72ca2 - jpountz - 2019-12-19 20:16:14]",
"at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:289) ~[lucene-core-8.4.0.jar:8.4.0 bc02ab906445fcf4e297f4ef00ab4a54fdd72ca2 - jpountz - 2019-12-19 20:16:14]",
"at org.apache.lucene.index.StandardDirectoryReader$1.doBody(StandardDirectoryReader.java:64) ~[lucene-core-8.4.0.jar:8.4.0 bc02ab906445fcf4e297f4ef00ab4a54fdd72ca2 - jpountz - 2019-12-19 20:16:14]",
"at org.apache.lucene.index.StandardDirectoryReader$1.doBody(StandardDirectoryReader.java:61) ~[lucene-core-8.4.0.jar:8.4.0 bc02ab906445fcf4e297f4ef00ab4a54fdd72ca2 - jpountz - 2019-12-19 20:16:14]",
"at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:680) ~[lucene-core-8.4.0.jar:8.4.0 bc02ab906445fcf4e297f4ef00ab4a54fdd72ca2 - jpountz - 2019-12-19 20:16:14]",
"at org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:84) ~[lucene-core-8.4.0.jar:8.4.0 bc02ab906445fcf4e297f4ef00ab4a54fdd72ca2 - jpountz - 2019-12-19 20:16:14]",
"at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:76) ~[lucene-core-8.4.0.jar:8.4.0 bc02ab906445fcf4e297f4ef00ab4a54fdd72ca2 - jpountz - 2019-12-19 20:16:14]",
"at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:64) ~[lucene-core-8.4.0.jar:8.4.0 bc02ab906445fcf4e297f4ef00ab4a54fdd72ca2 - jpountz - 2019-12-19 20:16:14]",
"at org.elasticsearch.gateway.PersistedClusterStateService.nodeMetaData(PersistedClusterStateService.java:262) ~[elasticsearch-7.6.2.jar:7.6.2]",
"at org.elasticsearch.env.NodeEnvironment.loadNodeMetaData(NodeEnvironment.java:408) ~[elasticsearch-7.6.2.jar:7.6.2]",
"at org.elasticsearch.env.NodeEnvironment.(NodeEnvironment.java:329) ~[elasticsearch-7.6.2.jar:7.6.2]",
"at org.elasticsearch.node.Node.(Node.java:277) ~[elasticsearch-7.6.2.jar:7.6.2]",
"at org.elasticsearch.node.Node.(Node.java:257) ~[elasticsearch-7.6.2.jar:7.6.2]",
"at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:221) ~[elasticsearch-7.6.2.jar:7.6.2]",
"at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:221) ~[elasticsearch-7.6.2.jar:7.6.2]",
"at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:349) ~[elasticsearch-7.6.2.jar:7.6.2]",
"at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) ~[elasticsearch-7.6.2.jar:7.6.2]",
"... 6 more"] }
ElasticsearchException[failed to bind service]; nested: CorruptIndexException[Unexpected file read error while reading index. (resource=BufferedChecksumIndexInput(SimpleFSIndexInput(path="/usr/share/elasticsearch/data/nodes/0/_state/segments_7ca0")))]; nested: NoSuchFileException[/usr/share/elasticsearch/data/nodes/0/_state/_78qd.si];
Likely root cause: java.nio.file.NoSuchFileException: /usr/share/elasticsearch/data/nodes/0/_state/_78qd.si
at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)
at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116)
at java.base/sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:219)
at java.base/java.nio.file.Files.newByteChannel(Files.java:374)
at java.base/java.nio.file.Files.newByteChannel(Files.java:425)
at org.apache.lucene.store.SimpleFSDirectory.openInput(SimpleFSDirectory.java:77)
at org.apache.lucene.store.Directory.openChecksumInput(Directory.java:157)
at org.apache.lucene.codecs.lucene70.Lucene70SegmentInfoFormat.read(Lucene70SegmentInfoFormat.java:91)
at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:353)
at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:289)
at org.apache.lucene.index.StandardDirectoryReader$1.doBody(StandardDirectoryReader.java:64)
at org.apache.lucene.index.StandardDirectoryReader$1.doBody(StandardDirectoryReader.java:61)
at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:680)
at org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:84)
at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:76)
at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:64)
at org.elasticsearch.gateway.PersistedClusterStateService.nodeMetaData(PersistedClusterStateService.java:262)
at org.elasticsearch.env.NodeEnvironment.loadNodeMetaData(NodeEnvironment.java:408)
at org.elasticsearch.env.NodeEnvironment.(NodeEnvironment.java:329)
at org.elasticsearch.node.Node.(Node.java:277)
at org.elasticsearch.node.Node.(Node.java:257)
at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:221)
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:221)
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:349)
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170)
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:161)
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86)
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:125)
at org.elasticsearch.cli.Command.main(Command.java:90)
<<>>
For complete error details, refer to the log at /usr/share/elasticsearch/logs/helk-cluster.log

@priamai
Copy link

priamai commented Apr 9, 2021

Try to increase the vmem size and then re run:

https://www.elastic.co/guide/en/elasticsearch/reference/current/vm-max-map-count.html

@SecOpsWarrior
Copy link
Author

I do that by default now as I ran into that issue a while back - current settings are:

Custom Options

#Fix ulimit in docker and snap
fs.file-max = 518144
vm.max_map_count = 262144

@SecOpsWarrior
Copy link
Author

I notice there is no longer a docker-compose.yml file in the HELK root directory on Github...did that get moved somewhere or did it accidentally get deleted?

@SecOpsWarrior
Copy link
Author

Would have liked to get this one figured out but I just nuked the whole thing and deployed a Debian server, installed a fresh version of Docker, and then ran install. Worked without a problem.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@SecOpsWarrior @priamai