Kibana dashboard not working with demo application

[ramesh-kothandapany]

Hi,
I am trying to install the demo project and try out various steps explained in the documentation.
I am able to successfully complete step 1 (Step 1: In Browser Widget)
With respect to step 2, i am not able to access Kibana dashboard

I am using docker toolbox in windows 7.
Executed below command and started all 3 services (elasticsearch, kibana, grafana)
docker-compose up -d
Executed docker ps
It listed as below

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
c72cebdf88d8 grafana/grafana:4.3.1 "/run.sh" 7 minutes ago Up 7 minutes 0.0.0.0:3000->3000/tcp springpetclinic_grafana4_1
ef32a4d3c251 docker.elastic.co/kibana/kibana:5.4.0 "/bin/sh -c /usr/loca" 7 minutes ago Up 7 minutes 0.0.0.0:5601->5601/tcp springpetclinic_kibana5_1
70451b53e6ac docker.elastic.co/elasticsearch/elasticsearch:5.4.0 "/bin/bash bin/es-doc" 7 minutes ago Up 7 minutes 0.0.0.0:9200->9200/tcp, 9300/tcp elasticsearch

Then i ran below command
mvn tomcat7:run

This command started the application in tomcat server.

When i executed below URL for elasticsearch, it works fine
http://192.168.99.100:9200/

But Kibana dashboard doesnt work when i execute below URL
http://192.168.99.100:5601/ (It says "this site can't be reached).

In docker-compose.yml, i checked config option for Kibana having attribute ELASTICSEARCH_URL
Original value of this is http://elasticsearch:9200
I changed it to http://localhost:9200, still didnt work
I changed it to http://192.168.99.100:9200, still didnt work
I changed it to http://172.18.0.2:9200, still didnt work

NOTE: When i execute docker-compose up, i can see logs appearing for Grafana and ES. But no logs i could see for Kibana.
log.txt

Any help on this will be great.

[felixbarny]

Could you try to execute docker-compose build before docker-compose up?

[ramesh-kothandapany]

$ docker-compose build
elasticsearch5 uses an image, skipping
kibana5 uses an image, skipping
grafana4 uses an image, skipping

Above is error, since i was relying on docker image.
i can download Kibana and do a build, but not able to download since i got trojan warning while downloading Kibana.

WARNING: ASG has detected a virus/PUS in this
file!
File has been dropped.
ASG Administrator:
Virus/PUS: "Suspicious:Trojan-Downloader.Script.Generic" found!
URL: https://artifacts.elastic.co/downloads/kibana/kibana-5.6.0-windows-x86.zip

Currently i am using composer file which is coming along with project bundle. It looks different from what i see in below (atleast versions used). Would trying with versions as given in below file will help?
https://github.com/stagemonitor/spring-petclinic/blob/master/docker-compose.yml

[felixbarny]

Yes, please try the current version. You can check it out with git fetch; git reset --hard origin/master

WARNING: ASG has detected a virus/PUS in this
file!
File has been dropped.
ASG Administrator:
Virus/PUS: "Suspicious:Trojan-Downloader.Script.Generic" found!
URL: https://artifacts.elastic.co/downloads/kibana/kibana-5.6.0-windows-x86.zip

Who emits that warning? I hope that this is a false positive.

[ramesh-kothandapany]

Sure, i will try with latest and post the updates.
With respect to virus warning, i have to check with my admins and my guess is also it should be false positive.

[ramesh-kothandapany]

I have executed docker-compose build before docker-compose up.
Still couldnt get to Kibana dashboard

[felixbarny]

what do the logs say?

[ramesh-kothandapany]

Below is log which could be resulting in this. May be i should change ?
stagemonitor.elasticsearch.url = http://192.168.99.100:9200

Detailed logs in below reference URL.
log2.txt

INFO  Stagemonitor - Measurement Session is initialized: [application=My Application] [instance=My Instance] [host=kothanda-bc]
WARN  ElasticsearchClient - Elasticsearch is not available. Stagemonitor won't try to send documents to Elasticsearch until it is available again.
INFO  CorePlugin - Not sending metrics to InfluxDB (url=null, interval=60s)
INFO  CorePlugin - Sending metrics to Elasticsearch ([http://localhost:9200]) every 10s
ERROR ElasticsearchClient - Error while updating kibana index pattern, definition = java.net.ConnectException: Connection refused: connect, pattern path = stage
monitor-metrics-kibana-index-pattern.json
INFO  AlertingPlugin - Using ElasticsearchIncidentRepository for storing incidents.
INFO  EhCachePlugin - Can't monitor EhCache as the CacheManager could not be found. Check the config for 'stagemonitor.ehcache.name'. It could also be possible
that cache manager is not initialized yet. Stagemonitor will try to initialize again when the application serves the first request.
INFO  Stagemonitor - # stagemonitor status
INFO  Stagemonitor - System information: Java 1.8.0_40 (Oracle Corporation) Windows 7 6.1
INFO  Stagemonitor - OK   - Agent attachment
INFO  Stagemonitor - OK   - AlertingPlugin (version 0.83.2)
INFO  Stagemonitor - OK   - CorePlugin (version 0.83.2)
INFO  Stagemonitor - OK   - EhCachePlugin (version 0.83.2)
INFO  Stagemonitor - FAIL - Elasticsearch (Elasticsearch is not available)
INFO  Stagemonitor - OK   - ElasticsearchTracingPlugin (version 0.83.2)
INFO  Stagemonitor - OK   - JdbcPlugin (version 0.83.2)
INFO  Stagemonitor - OK   - JvmPlugin (version 0.83.2)
INFO  Stagemonitor - OK   - LoggingPlugin (version 0.83.2)
INFO  Stagemonitor - OK   - OsPlugin (version 0.83.2)
INFO  Stagemonitor - OK   - ServletPlugin (version 0.83.2)
INFO  Stagemonitor - OK   - SoapTracingPlugin (version 0.83.2)
INFO  Stagemonitor - OK   - Startup
INFO  Stagemonitor - OK   - TracingPlugin (version 0.83.2)

[ramesh-kothandapany]

i changed stagemonitor.elasticsearch.url = http://192.168.99.100:9200
Below is the log now, i could see some errors even though it can connect to ES.

INFO  Stagemonitor - Measurement Session is initialized: [application=My Application] [instance=My Instance] [host=kothanda-bc]
WARN  HttpClient$ErrorLoggingResponseHandler - http://192.168.99.100:9200/.kibana: 400 {"error":{"root_cause":[{"type":"index_already_exists_exception","reason"
:"index [.kibana/yLqbLJeDS_WC0zxaHN_ZGw] already exists","index_uuid":"yLqbLJeDS_WC0zxaHN_ZGw","index":".kibana"}],"type":"index_already_exists_exception","reas
on":"index [.kibana/yLqbLJeDS_WC0zxaHN_ZGw] already exists","index_uuid":"yLqbLJeDS_WC0zxaHN_ZGw","index":".kibana"},"status":400}
WARN  HttpClient$ErrorLoggingResponseHandler - http://192.168.99.100:9200/.kibana/_mapping/search: 400 {"error":{"root_cause":[{"type":"illegal_argument_excepti
on","reason":"mapper [hits] cannot be changed from type [long] to [integer]"}],"type":"illegal_argument_exception","reason":"mapper [hits] cannot be changed fro
m type [long] to [integer]"},"status":400}
WARN  HttpClient$ErrorLoggingResponseHandler - http://192.168.99.100:9200/.kibana/_mapping/dashboard: 400 {"error":{"root_cause":[{"type":"illegal_argument_exce
ption","reason":"mapper [refreshInterval.value] cannot be changed from type [long] to [integer]"}],"type":"illegal_argument_exception","reason":"mapper [refresh
Interval.value] cannot be changed from type [long] to [integer]"},"status":400}
WARN  HttpClient$ErrorLoggingResponseHandler - http://192.168.99.100:9200/.kibana/_mapping/visualization: 400 {"error":{"root_cause":[{"type":"illegal_argument_
exception","reason":"mapper [version] cannot be changed from type [long] to [integer]"}],"type":"illegal_argument_exception","reason":"mapper [version] cannot b
e changed from type [long] to [integer]"},"status":400}
INFO  CorePlugin - Not sending metrics to InfluxDB (url=null, interval=60s)
INFO  CorePlugin - Sending metrics to Elasticsearch ([http://192.168.99.100:9200]) every 10s
INFO  EhCachePlugin - Can't monitor EhCache as the CacheManager could not be found. Check the config for 'stagemonitor.ehcache.name'. It could also be possible
that cache manager is not initialized yet. Stagemonitor will try to initialize again when the application serves the first request.
INFO  AlertingPlugin - Using ElasticsearchIncidentRepository for storing incidents.
INFO  Stagemonitor - # stagemonitor status
INFO  Stagemonitor - System information: Java 1.8.0_40 (Oracle Corporation) Windows 7 6.1
INFO  Stagemonitor - OK   - Agent attachment
INFO  Stagemonitor - OK   - AlertingPlugin (version 0.83.2)
INFO  Stagemonitor - OK   - CorePlugin (version 0.83.2)
INFO  Stagemonitor - OK   - EhCachePlugin (version 0.83.2)
INFO  Stagemonitor - OK   - Elasticsearch
INFO  Stagemonitor - OK   - ElasticsearchTracingPlugin (version 0.83.2)
INFO  Stagemonitor - OK   - JdbcPlugin (version 0.83.2)
INFO  Stagemonitor - OK   - JvmPlugin (version 0.83.2)
INFO  Stagemonitor - OK   - LoggingPlugin (version 0.83.2)
INFO  Stagemonitor - OK   - OsPlugin (version 0.83.2)
INFO  Stagemonitor - OK   - ServletPlugin (version 0.83.2)
INFO  Stagemonitor - OK   - SoapTracingPlugin (version 0.83.2)
INFO  Stagemonitor - OK   - Startup
INFO  Stagemonitor - OK   - TracingPlugin (version 0.83.2)

[felixbarny]

Logs are looking good. You can safely ignore the errors.

Can you open kibana now? If not, could you post the docker-compose logs?

[ramesh-kothandapany]

http://192.168.99.100:5601/ -> this results in "This site can’t be reached" error in browser

service containers (ES, Kibana) is running in background. so i dont have access to logs. Is there anyway we could get logs when it is running in background. Below is the sequence i execute these commands.

docker-compose build
docker-compose up -d
mvn tomcat7:run

Will running 'docker-compose up' and sending logs will be helpful?

[felixbarny]

You can run docker-compose up to not run it in the background or execute docker-compose logs to see the logs

Will running 'docker-compose up' and sending logs will be helpful?

yes

[ramesh-kothandapany]

logs attached
log4.txt

[felixbarny]

It's really strange that there are no outputs from Kibana whatsoever. I can't quite explain that. I'm not using Windows at work, but I will try to reproduce that at home.

[ramesh-kothandapany]

I have seen Kibana logs, but it was on 14th Sept. Today, i had the same version of code/tool/software and nothing changed and dont see Kibana logs anymore.

log5.txt

I did pulled the latest of demo application code as well today, but still i couldnt see anything showing up for Kibana.

Thanks a lot Felix for your help. Let me know if you get any clue on what i could be missing.

[ramesh-kothandapany]

Please see below as continuation of log4.txt. After an hour, i have got below information in logs

←[33mgrafana4_1        |←[0m t=2017-09-18T12:26:23+0000 lvl=info msg="Initializing HTTP Server" logger=http.server address=0.0.0.0:3000 protocol=http subUrl= so
cket=
←[36melasticsearch     |←[0m [2017-09-18T13:31:36,932][WARN ][o.e.m.j.JvmGcMonitorService] [es-monitoring-01] [gc][young][3846][150] duration [9.8s], collection
s [1]/[9.8s], total [9.8s]/[13.3s], memory [231.1mb]->[164.1mb]/[503.6mb], all_pools {[young] [66.5mb]->[15.2kb]/[66.5mb]}{[survivor] [8.3mb]->[7.4mb]/[8.3mb]}{
[old] [156.2mb]->[156.7mb]/[428.8mb]}
←[36melasticsearch     |←[0m [2017-09-18T13:31:36,948][WARN ][o.e.m.j.JvmGcMonitorService] [es-monitoring-01] [gc][3846] overhead, spent [9.8s] collecting in th
e last [9.8s]
←[32mkibana5_1         |←[0m {"type":"log","@timestamp":"2017-09-18T13:17:40Z","tags":["info","optimize"],"pid":1,"message":"Optimizing and caching bundles for
stagemonitor-kibana, kibana, stateSessionStorageRedirect, timelion and status_page. This may take a few minutes"}
←[32mspringpetclinic_kibana5_1 exited with code 137
←[0m

[ramesh-kothandapany]

Removed below section in docker compose. I am able to access the Kibana dashboard now. Thanks for your help. I will continue to learn this and post if i have any further queries.

    environment:
     - XPACK_GRAPH_ENABLED=false
     - XPACK_MONITORING_ENABLED=false
     - XPACK_ML_ENABLED=false
     - XPACK_REPORTING_ENABLED=false
     - XPACK_SECURITY_ENABLED=false
     - XPACK_WATCHER_ENABLED=false
     - ELASTICSEARCH_URL=http://elasticsearch:9200