Kibana status RED using v1.9.1

[johncst]

Just installed v1.9.1 with ES 2.3.1, and plugged Use Case #2 into elasticsearch.yml. ES started OK, but starting the Kibana server showed:
~/kibana$ bin/kibana
log [09:44:26.639] [info][status][plugin:kibana] Status changed from uninitialized to green - Ready
log [09:44:26.689] [info][status][plugin:elasticsearch] Status changed from uninitialized to yellow - Waiting for Elasticsearch
log [09:44:26.715] [info][status][plugin:kbn_vislib_vis_types] Status changed from uninitialized to green - Ready
log [09:44:26.724] [info][status][plugin:markdown_vis] Status changed from uninitialized to green - Ready
log [09:44:26.729] [info][status][plugin:metric_vis] Status changed from uninitialized to green - Ready
log [09:44:26.736] [info][status][plugin:spyModes] Status changed from uninitialized to green - Ready
log [09:44:26.754] [info][status][plugin:statusPage] Status changed from uninitialized to green - Ready
log [09:44:26.760] [info][status][plugin:table_vis] Status changed from uninitialized to green - Ready
log [09:44:26.783] [info][listening] Server running at http://0.0.0.0:5601
log [09:44:26.792] [error][status][plugin:elasticsearch] Status changed from yellow to red - Authentication Exception

Trying to start Kibana from the browser showed a RED - authentication error message. I restarted ES, and it became unresponsive. Any request to :9200 waited forever. Removing the plugin broke ES and Kibana to the point they had to be reloaded.

If I can provide any additional debugging info, please let me know. This might be user error, but I thought I'd let you know...

[sscarduzio]

Hi @johncst, did you configure kibana.yml with the HTTP Basic Auth credentials? Please post the relevant parts of your elasticsearch.yml and kibana.yml so we can investigate better!

[johncst]

Hi @sscarduzio, thanks for the reply. All I did was what it said in the readme.md, which was put case #2 in elasticsearch.yml, which I now see is probably my issue. I didn't put anything in kibana.yml. I saw the snippet in your example directory, so I'm guessing I need to alter what I did in both .yml files.
elasticsearch.yml -
readonlyrest:
enable: true
response_if_req_forbidden:

Forbidden

access_control_rules:

- name: Salesmen (read only)
  type: allow
  kibana_access: ro
  auth_key: sales:passwd1

- name: Managers (read only, but can create dashboards)
  type: allow
  kibana_access: ro+
  auth_key: manager:passwd2

- name: Admin (read write)
  type: allow
  kibana_access: rw
  auth_key: admin:passwd3

kibana.yml -
(nothing)

If you have any additional guidelines on what goes into elasticsearch.yml and what goes into kibana.yml, I'd appreciate the advice and instruction. Like I said, I figured it was user error on my part. I just don't know enough about your plugin yet. However BRAVO for doing this and sharing with the rest of us!!!

[sscarduzio]

Hey @johncst, thanks for the compliment. My pleasure! 👍

Yes, I believe the instructions are a bit incomplete then. Thanks for finding out!
Please add this to your kibana.yml (or uncomment existing).

elasticsearch.username: "admin"
elasticsearch.password: "passwd3"

This will make sure that the kibana process is free to talk to ES, however, the browser will still be required to insert the username and password.

BTW: would you like to contribute to the project by adding the solution to your issue to the readme? I don't want to steal the credits :)

Enjoy!

[johncst]

No credit necessary. I'm more than happy to add my very small contribution when people like you contribute your time to help others. I suggest adding the following after USE CASE 2, and before ##### 3 - Restart Elasticsearch. BTW, I tried this under both Ubuntu and Windows, and it works in both.

If you are using authentication (USE CASE 2), you need to activate authentication in kibana.yml as well. Authentication in Kibana is not active by default, and if you look in kibana.yml, you will see the default username and password commented-out:

# If your Elasticsearch is protected with basic authentication, these settings provide
# the username and password that the Kibana server uses to perform maintenance on the Kibana
# index at startup. Your Kibana users still need to authenticate with Elasticsearch, which 
# is proxied through the Kibana server.
# elasticsearch.username: "user"
# elasticsearch.password: "pass"

Un-comment the elasticsearch.username and elasticsearch.password lines, and change the entries to match those found in elasticsearch.yml. Referring to USE CASE 2, the entries would now look like:

# If your Elasticsearch is protected with basic authentication, these settings provide
# the username and password that the Kibana server uses to perform maintenance on the Kibana
# index at startup. Your Kibana users still need to authenticate with Elasticsearch, which 
# is proxied through the Kibana server.
elasticsearch.username: "admin"
elasticsearch.password: "passwd3"

Restart the Kibana server.

[rocky4bmw]

Hi sscarduzio,

I am using Kibana 4.5.1 , ES of 2.3, shiled of 2.3.3 but when I tried to start Kibana I get below error

"[error][status][plugin:elasticsearch] Status changed from yellow to red - Authentication Exception"

So in kibana.yml file I have uncommented below lines after creating euser from commandline with admin role

elasticsearch.url: "http//localhost:9200"
elasticsearch.username: "xxx"
elasticsearch.password: "xxx"

Now I am getting below error "ValidationError: child "elasticsearch" fails because [child "url" fails because ["url" must be a valid uri with a scheme matching the http|https pattern]]"

Can you please help here to fix the issue?

[sscarduzio]

You missed the semicolon before the double slash!

[rocky4bmw]

Thanks a lot for your reply. It worked fine.But when I logged into Kibana it gives message as

Index Patterns
Warning No default index pattern. You must select or create one to continue

I tried to download filebeat template I get below error

curl -XPUT 'http://ipaddress/_template/filebeat?pretty' -d@filebeat-index-template.json
{
"error" : {
"root_cause" : [ {
"type" : "security_exception",
"reason" : "missing authentication token for REST request [/_template/filebeat?pretty]",
"header" : {
"WWW-Authenticate" : "Basic realm="shield""
}
} ],
"type" : "security_exception",
"reason" : "missing authentication token for REST request [/_template/filebeat?pretty]",
"header" : {
"WWW-Authenticate" : "Basic realm="shield""
}
},
"status" : 401
}

[rocky4bmw]

curl -XPUT 'http://localhost:9200/_template/filebeat?pretty' -d@filebeat-index-template.json
Warning: Couldn't read data from file "filebeat-index-template.json", this
Warning: makes an empty POST.

[sscarduzio]

Well, if you're trying to create stuff via the REST API in an instance of ElasticSearch with this plugin installed, you'll get a 401 because that's how it is supposed to work. Right?

You have to disable the plugin first, create the template and re-enable it. At least this is how I understood your situation.

[sscarduzio]

Also, pass the absolute path of the json file to that curl.

[rocky4bmw]

@sscarduzio

Let me give a quick view what exactly I am trying to do

Step1:

I have installed Elastic search 2.3 , Kibana 4.5, Logastash 2.3 and shield 2.3 on single server.

In elasticsearch.yml I have uncommented

#network.host: private ip

In Kibana.yml I have uncommented below

#server.host: "privateip"

#elasticsearch.url: "http://privateip:9200"

#elasticsearch.username: "xxx"
#elasticsearch.password: "xxx"

#server.ssl.cert: absolutepath of certifcate
#server.ssl.key: absolutepath of key
#shield.encryptionKey: 'xxx'

created using esusers command line I have created one admin user with admin role and another user with user role.

Right now I am able to login to Kibana with user credentials created.

Step 2:

Load Kibana Dashboards

curl -L -O https://download.elastic.co/beats/dashboards/beats-dashboards-1.1.0.zip
unzip beats-dashboards-.zip
cd beats-dashboards-
./load.sh

_It fails_***************

Load Filebeat Index Template in Elasticsearch in home directory of ELK server

curl -O https://gist.githubusercontent.com/thisismitch/3429023e8438cc25b86c/raw/d8c479e2a1adcea8b1fe86570e42abab0f10f364/filebeat-index-template.json

curl -XPUT 'http://localhost:9200/_template/filebeat?pretty' -d@filebeat-index-template.json

Above setps fails *************************

Step 3: Setup Filebeat on client servers

Copy SSL Certificate from ELK server to absolute path of client server

Install Filebeat package

filebeat.yml is as below

paths:
- /var/log/*.log

Elasticsearch as output

elasticsearch:

 hosts: ["privateip:9200"]

Path to template file

 path: "filebeat.template.json"

Logstash as output

logstash:
# The Logstash hosts
hosts: ["privateip:5044"]
bulk_max_size: 1024

Optional TLS. By default is off.

tls:
# List of root certificates for HTTPS server verifications
certificate_authorities: ["//absoluepath of certifcate"]

Ste 4: Test Filebeat Installation

curl -XGET 'http://localhost:9200/filebeat-*/_search?pretty'

It fails

Can you please let me know how can I proceed. For security purpose I have installed Shield I am facing lot of issues.

With NGINX it is straight forward but there is no control on user access.

[sscarduzio]

Wait, you're using Shield? So you're not using readonlyrest plugin?

[rocky4bmw]

yes thats true I am using Shiled now. How can I create indexing using filebeat and load Kibana dashboards.

[sscarduzio]

Well, I'd love to help you, but I never used Shield. The reason is that It's proprietary software and I'm not interested in using it. This plugin pre-dates Shield and covers just a few of the features of Shield, and they work very differently!

[rocky4bmw]

do you have any links where it gives steps on how to install shield or can I still use REST API.

As per below links of elasticsearch websites still they use REST API

https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-installation.html

[rocky4bmw]

@sscarduzio

I fiugured out few things like passing on user details and I am at last step of indexing which failed. Can you help here please

Step1:

I have installed Elastic search 2.3 , Kibana 4.5, Logastash 2.3 and shield 2.3 on single server.

created using esusers command line I have created one admin user with admin role and another user with user role.

Right now I am able to login to Kibana with user credentials created.

Step 2:

Loaded Kibana Dashboards
Load Filebeat Index Template in Elasticsearch
installed filebeat and configured it on client server.

step3 : Tried to test filebeat installation and I get below error.

curl -XGET "http://ipaddress:9200/filebeat-_/_search?pretty" -u admin -p
Enter host password for user 'admin':xxx
{
"error" : {
"root_cause" : [ {
"type" : "index_not_found_exception",
"reason" : "no such index",
"index" : "[filebeat-_]"
} ],
"type" : "index_not_found_exception",
"reason" : "no such index",
"index" : "[filebeat-*]"
},
"status" : 404
}

Please help me in fixing this issue

[rocky4bmw]

Just addidtional info which can help you

After loginpage opens up in Kibana. I did below steps

Loading Kibana dashboards:

curl -L -O http://download.elastic.co/beats/dashboards/beats-dashboards-1.2.3.zip

unzip beats-dashboards-1.2.3.zip
cd beats-dashboards-1.2.3/

./load.sh -url http://ipaddress:9200 -u admin:xxx

_Load Filebeat Index Template in Elasticsearch_*******************

curl -L -O https://gist.githubusercontent.com/thisismitch/3429023e8438cc25b86c/raw/d8c479e2a1adcea8b1fe86570e42abab0f10f364/filebeat-index-template.json

_Load the template with this command:_****************************

curl -XPUT 'http://ipaddress:9200/_template/filebeat?pretty' -d@filebeat-index-template.json

_Installed filebeat and configured it on client server_*********************************

step3 : Tried to test filebeat installation on ELK server and I get below error.

curl -XGET "http://ipaddress:9200/filebeat-_/_search?pretty" -u admin -p
Enter host password for user 'admin':xxx
{
"error" : {
"root_cause" : [ {
"type" : "index_not_found_exception",
"reason" : "no such index",
"index" : "[filebeat-_]"
} ],
"type" : "index_not_found_exception",
"reason" : "no such index",
"index" : "[filebeat-*]"
},
"status" : 404
}

Please help me in fixing this issue