Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Can't get Elk started #364

Closed
YouveGotMeowxy opened this issue Mar 21, 2022 · 7 comments
Closed

Can't get Elk started #364

YouveGotMeowxy opened this issue Mar 21, 2022 · 7 comments

Comments

@YouveGotMeowxy
Copy link

I'm using Docker Desktop in WSL2 and for the life of me I can't get it running, can anyone tell me what's wrong?

My compose:

services:

  elk:
    hostname: elk
    image: sebp/elk:latest
    environment:
      TZ: 'America/Chicago'
      ES_CONNECT_RETRY: '60'
      CLUSTER_NAME: 'cloud-cluster'
    volumes:
        - /opt/docker/configs/elk/kibana:/opt/kibana/config:rw     # Kibana config
        - /opt/docker/configs/elasticsearch:/etc/elasticsearch:rw   # Elasticsearch config         
        - /mnt/e/Docker/elk:/var/lib/elasticsearch:rw                      # Elasticsearch data
        - /mnt/e/Docker/Logs/elk:/var/log:rw                                 # Logs
    #ports: (I'm doing everything internally within docker's network)
        #- '9034:5601' # Kibana
        #- '9036:5044' # Logstash
        #- '9035:9200' # Elasticsearch (JSON Interface)
        #- '9300:9300' # Elasticsearch (Transport Interface: Optional)
    networks:
      - elkn
    deploy:
      mode: replicated
      replicas: 1
      placement:
        constraints:
          - node.labels.MainDaemon == true
      restart_policy:
        condition: any

networks:
  elkn:
    external: true
    name: monitors

First thing that happens straight away is this:

today at 11:22:38 AM * Starting periodic command scheduler cron
today at 11:22:38 AM   ...done.
today at 11:22:38 AM * Starting Elasticsearch Server
today at 11:22:40 AM**Exception in thread "main" java.nio.file.NoSuchFileException: /etc/elasticsearch/jvm.options**
today at 11:22:40 AM	at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)
today at 11:22:40 AM	at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)
today at 11:22:40 AM	at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
today at 11:22:40 AM	at java.base/sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:219)
today at 11:22:40 AM	at java.base/java.nio.file.Files.newByteChannel(Files.java:380)
today at 11:22:40 AM	at java.base/java.nio.file.Files.newByteChannel(Files.java:432)
today at 11:22:40 AM	at java.base/java.nio.file.spi.FileSystemProvider.newInputStream(FileSystemProvider.java:422)
today at 11:22:40 AM	at java.base/java.nio.file.Files.newInputStream(Files.java:160)
today at 11:22:40 AM	at org.elasticsearch.tools.launchers.JvmOptionsParser.readJvmOptionsFiles(JvmOptionsParser.java:168)
today at 11:22:40 AM	at org.elasticsearch.tools.launchers.JvmOptionsParser.jvmOptions(JvmOptionsParser.java:124)
today at 11:22:40 AM	at org.elasticsearch.tools.launchers.JvmOptionsParser.main(JvmOptionsParser.java:86)
today at 11:22:40 AM   ...fail!
today at 11:22:41 AMwaiting for Elasticsearch to be up (1/60)
today at 11:22:42 AMwaiting for Elasticsearch to be up (2/60)
today at 11:22:43 AMwaiting for Elasticsearch to be up (3/60)
today at 11:22:44 AMwaiting for Elasticsearch to be up (4/60)
today at 11:22:45 AMwaiting for Elasticsearch to be up (5/60)
today at 11:22:46 AMwaiting for Elasticsearch to be up (6/60)
today at 11:22:47 AMwaiting for Elasticsearch to be up (7/60)
today at 11:22:48 AMwaiting for Elasticsearch to be up (8/60)
today at 11:22:49 AMwaiting for Elasticsearch to be up (9/60)
today at 11:22:50 AMwaiting for Elasticsearch to be up (10/60)
today at 11:22:51 AMwaiting for Elasticsearch to be up (11/60)
today at 11:22:52 AMwaiting for Elasticsearch to be up (12/60)
today at 11:22:53 AMwaiting for Elasticsearch to be up (13/60)
today at 11:22:54 AMwaiting for Elasticsearch to be up (14/60)
today at 11:22:55 AMwaiting for Elasticsearch to be up (15/60)
today at 11:22:56 AMwaiting for Elasticsearch to be up (16/60)
today at 11:22:57 AMwaiting for Elasticsearch to be up (17/60)
today at 11:22:58 AMwaiting for Elasticsearch to be up (18/60)
today at 11:22:59 AMwaiting for Elasticsearch to be up (19/60)
today at 11:23:00 AMwaiting for Elasticsearch to be up (20/60)
today at 11:23:01 AMwaiting for Elasticsearch to be up (21/60)
today at 11:23:02 AMwaiting for Elasticsearch to be up (22/60)
today at 11:23:03 AMwaiting for Elasticsearch to be up (23/60)
today at 11:23:04 AMwaiting for Elasticsearch to be up (24/60)
today at 11:23:05 AMwaiting for Elasticsearch to be up (25/60)
today at 11:23:06 AMwaiting for Elasticsearch to be up (26/60)
today at 11:23:07 AMwaiting for Elasticsearch to be up (27/60)
today at 11:23:08 AMwaiting for Elasticsearch to be up (28/60)
today at 11:23:09 AMwaiting for Elasticsearch to be up (29/60)
today at 11:23:10 AMwaiting for Elasticsearch to be up (30/60)
today at 11:23:11 AMwaiting for Elasticsearch to be up (31/60)
today at 11:23:12 AMwaiting for Elasticsearch to be up (32/60)
today at 11:23:13 AMwaiting for Elasticsearch to be up (33/60)
today at 11:23:14 AMwaiting for Elasticsearch to be up (34/60)
today at 11:23:15 AMwaiting for Elasticsearch to be up (35/60)
today at 11:23:16 AMwaiting for Elasticsearch to be up (36/60)
today at 11:23:17 AMwaiting for Elasticsearch to be up (37/60)
today at 11:23:18 AMwaiting for Elasticsearch to be up (38/60)
today at 11:23:19 AMwaiting for Elasticsearch to be up (39/60)
today at 11:23:20 AMwaiting for Elasticsearch to be up (40/60)
today at 11:23:21 AMwaiting for Elasticsearch to be up (41/60)
today at 11:23:22 AMwaiting for Elasticsearch to be up (42/60)
today at 11:23:23 AMwaiting for Elasticsearch to be up (43/60)
today at 11:23:24 AMwaiting for Elasticsearch to be up (44/60)
today at 11:23:25 AMwaiting for Elasticsearch to be up (45/60)
today at 11:23:26 AMwaiting for Elasticsearch to be up (46/60)
today at 11:23:27 AMwaiting for Elasticsearch to be up (47/60)
today at 11:23:28 AMwaiting for Elasticsearch to be up (48/60)
today at 11:23:29 AMwaiting for Elasticsearch to be up (49/60)
today at 11:23:30 AMwaiting for Elasticsearch to be up (50/60)
today at 11:23:31 AMwaiting for Elasticsearch to be up (51/60)
today at 11:23:32 AMwaiting for Elasticsearch to be up (52/60)
today at 11:23:33 AMwaiting for Elasticsearch to be up (53/60)
today at 11:23:34 AMwaiting for Elasticsearch to be up (54/60)
today at 11:23:35 AMwaiting for Elasticsearch to be up (55/60)
today at 11:23:36 AMwaiting for Elasticsearch to be up (56/60)
today at 11:23:37 AMwaiting for Elasticsearch to be up (57/60)
today at 11:23:38 AMwaiting for Elasticsearch to be up (58/60)
today at 11:23:39 AMwaiting for Elasticsearch to be up (59/60)
today at 11:23:40 AMwaiting for Elasticsearch to be up (60/60)
today at 11:23:40 AMCouldn't start Elasticsearch. Exiting.
today at 11:23:40 AMElasticsearch log follows below.
today at 11:23:41 AMContainer stopped

And then nothing gets written to any of the mounted volumes, or the log. No config files, nothing. I checked permissions for the config directories, and they are all root (is that correct?).

I'm not very well versed in linux and am still learning. Shouldn't the config files, including the jvm.options file automatically get written/copied to the mounted volumes?

When I followed your instructions and did this:

sudo docker run -it sebp/elk:latest bash

and:

ES_PATH_CONF=/etc/elasticsearch gosu elasticsearch /opt/elasticsearch/bin/elasticsearch \
    -Epath.logs=/var/log/elasticsearch \
    -Epath.data=/var/lib/elasticsearch

it seems to run fine, so I'm thinking something is wrong with my compose? But I cant sort it out. Can anyone help me?
@spujadas
Copy link
Owner

My first thought is that the local volumes that you’re bind-mounting could be missing files that are needed to start the services.

Specifically, this error message

today at 11:22:40 AM**Exception in thread "main" java.nio.file.NoSuchFileException: /etc/elasticsearch/jvm.options**

makes this line of your compose suspicious:

    - /opt/docker/configs/elasticsearch:/etc/elasticsearch:rw   # Elasticsearch config

In the default image, the /etc/elasticsearch directory contains quite a few configuration files:

root@d546cc5e58c3:/etc/elasticsearch# ll
total 52
drwxr-xr-x 1 elasticsearch elasticsearch  4096 Mar 21 19:28 ./
drwxr-xr-x 1 root          root           4096 Mar 21 19:28 ../
-rw-rw---- 1 elasticsearch elasticsearch   199 Mar 21 19:28 elasticsearch.keystore
-rwxrwxrwx 1 elasticsearch elasticsearch  2900 Jan  5 19:26 elasticsearch.yml*
-rw-r--r-- 1 elasticsearch elasticsearch  3257 Mar 21 19:28 jvm.options
drwxr-xr-x 1 elasticsearch elasticsearch  4096 Mar 21 19:27 jvm.options.d/
-rw-r--r-- 1 elasticsearch elasticsearch 19150 Mar 21 19:28 log4j2.properties
drwxr-xr-x 1 elasticsearch elasticsearch  4096 Mar 21 19:27 scripts/

I’m guessing that your host’s /opt/docker/configs/elasticsearch is missing (at least) one these files.
If you only want to override Elasticsearch’s elasticsearch.yml config file whilst keeping all the other files as is, then I’d suggest changing your config to something like:

    - /opt/docker/configs/elasticsearch/elasticsearch.yml:/etc/elasticsearch/elasticsearch.yml:rw   # Elasticsearch config

If nothing else, at least it should sort out the very first error message that you’re seeing 😄

@spujadas
Copy link
Owner

Shouldn't the config files, including the jvm.options file automatically get written/copied to the mounted volumes?

Sorry, missed your question. The way Docker’s volume mounting system works, it’s actually the other way round, so to speak: the host’s files/directories replace the ones in the image.

@YouveGotMeowxy
Copy link
Author

YouveGotMeowxy commented Mar 22, 2022 •
edited
Loading

@spujadas TY, I got it up! : D

I don't know why, but I had already tried that with the standalone ES container and it wouldn't work. I decided to try your ELK thinking it was all already gonna work out of the box, and copy basic files into my mounted folders and i could go from there (it didnt, as per my original question). I copied the same es.yml I was already using and mounted it in elk ... and it worked! lol

I just have a couple of quick questions now, if you dont mind?

  • What if we want to upgrade ES to the latest 8.1.0? (Normally in standalone containers we can easily update but just pulling the new container; I'm not sure how it works in ELK?)
  • Same question for any of the others, if ever needed? (kibana, logstash)
  • How should I go about mounting my personal kibana.yml? Is there a path inside the running elk that I should copy it out of first, and then mount it with my tweaks?
  • *I got this while visiting one of the pages in kibaba, is this a big deal?

image

I apologize for so many questions, but I'm new to all of this, and I figure you know this stuff like the back of your hand so the questions would be easy. :)

*I found this, so the last question is answered :)

@spujadas
Copy link
Owner

Glad to hear you got it working 👍

What if we want to upgrade ES to the latest 8.1.0? (Normally in standalone containers we can easily update but just pulling the new container; I'm not sure how it works in ELK?)

Same for ELK, you can just pull sebp/elk:xyz where xyz is the tag of one of the pre-built versions of the image (see https://github.com/spujadas/elk-docker or https://hub.docker.com/r/sebp/elk for the complete list).
As it turns out I haven’t built the image since February so version 8.1.0 isn’t available yet, but I’ll look into it shortly.

Same question for any of the others, if ever needed? (kibana, logstash)

Since version 5, each image contains the same version of Elasticsearch, Logstash, and Kibana, so same answer as above.

How should I go about mounting my personal kibana.yml? Is there a path inside the running elk that I should copy it out of first, and then mount it with my tweaks?

In the image, Kibana’s config file is ${KIBANA_HOME}/config/kibana.yml, which resolves to /opt/kibana/config/kibana.yml.
Instead of copying it out of the image (which would work, of course), you could copy/download the source file from the GitHub repo (https://github.com/spujadas/elk-docker/blob/master/kibana.yml), which is used to build the image.

  • *I got this while visiting one of the pages in kibaba, is this a big deal?

The link that you found points to my outdated ELKX image, so not sure if that will actually help you.
What you’re seeing is not a big deal if you don’t need security. If you do want to use security features, then you can just follow the instructions in the message and tweak the security configuration to your heart’s content 😉

@YouveGotMeowxy
Copy link
Author

OK, again, TYVM for the help!

I wasn't sure of copying the kibana.yml from the repo, since I tried that with the jv.options file while messing with the standalone ES, and got errors, because I think the file gets some values written to it when it starts running and the "raw" file only has tokens as placeholders in it.

Yes, my next step is to add security. I also had issues trying to do THAT in standalone, due to having to mount the .keystore, and then persimmons for it, and ... on and on and on .... ugggghhh, lol.

@spujadas
Copy link
Owner

Yep, setting up security is definitely not for the faint of heart :laugh:

I’ve just updated the repository and the image with the latest version of the ELK stack, namely 8.1.0, if you want to give it a spin.

(And closing this issue.)

@YouveGotMeowxy
Copy link
Author

Thank you!

I just upgraded and so far, so good!

I had a couple small issues when using the latest tag, like:

java.lang.IllegalStateException: cannot upgrade a node from version [7.16.3] directly to version [8.1.0], upgrade to version [7.17.0] first.

but just doing a quick restart using the 7.17.1 tag, and then back to the latest tag solved that.

and then I got this error:

ERROR: [1] bootstrap checks failed. You must address the points described in the following [1] lines before starting Elasticsearch.
today at 3:38:15 PMbootstrap check failure [1] of [1]: The default value for [xpack.security.enabled] has changed in the current version. Security features were implicitly disabled for this node but they would now be enabled, possibly preventing access to the node. See https://www.elastic.co/guide/en/elasticsearch/reference/8.1/security-minimal-setup.html to configure security, or explicitly disable security by setting [xpack.security.enabled] to "false" in elasticsearch.yml before restarting the node.
today at 3:38:15 PMERROR: Elasticsearch did not exit normally - check the logs at /var/log/domain/domain-cloud.log

But for now (until I wrestle with security) manually setting it to false in the config solved it.

Then, I was getting a page full of these in the log:

[2022-03-23T15:40:12,783][WARN ][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"http://localhost:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError, :message=>"Got response code '401' contacting Elasticsearch at URL 'http://localhost:9200/'"}

No idea what that is or was, but it seems to have sorted itself out (at least for now; no idea if it'll show up again at some point)

Other than that, everything seems to be going great, ty for maintaining this! :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@spujadas @YouveGotMeowxy