Unable to see Watchers in the Sentinl Watcher Table #469

aashish051218 · 2018-06-20T12:48:03Z

I am using ES version 6.2.3, Kibana version 6.2.2
Issue description
Even though it says that the watcher is successfully created, I still do not see the same in the watcher table
Reproducing the issue step-by-step

I was expecting a watcher table something like this

Can somebody please help me out?

lmangani · 2018-06-20T13:01:03Z

Could you confirm if you are using any authentication mechanism? Also grab the Kibana logs as you save the watcher, they should reveal some sort of error response while attempting to store data.

aashish051218 · 2018-06-20T13:53:47Z

Hi,
Thanks for the response.
I created a watcher named - test. Here is the log. There is no authentication mechanism.

"{"type":"log","@timestamp":"2018-06-20T13:49:18Z","tags":["warning"],"pid":15274,"kibanaVersion":"6.2.2","nodes":[{"version":"6.2.3","http":{"publish
_address":"192.168.10.69:6968"},"ip":"192.168.10.69"},{"version":"6.2.3","http":{"publish_address":"192.168.10.121:6968"},"ip":"192.168.10.121"},{"ve
rsion":"6.2.3","http":{"publish_address":"192.168.10.182:6968"},"ip":"192.168.10.182"},{"version":"6.2.3","http":{"publish_address":"192.168.10.116:6
968"},"ip":"192.168.10.116"},{"version":"6.2.3","http":{"publish_address":"192.168.10.184:6968"},"ip":"192.168.10.184"},{"version":"6.2.3","http":{"p
ublish_address":"192.168.10.117:6968"},"ip":"192.168.10.117"},{"version":"6.2.3","http":{"publish_address":"192.168.10.183:6968"},"ip":"192.168.10.18
3"},{"version":"6.2.3","http":{"publish_address":"192.168.10.143:6968"},"ip":"192.168.10.143"}],"message":"You're running Kibana 6.2.2 with some diff
erent versions of Elasticsearch. Update Kibana or Elasticsearch to the same version to prevent compatibility issues: v6.2.3 @ 192.168.10.69:6968 (192
.168.10.69), v6.2.3 @ 192.168.10.121:6968 (192.168.10.121), v6.2.3 @ 192.168.10.182:6968 (192.168.10.182), v6.2.3 @ 192.168.10.116:6968 (192.168.10.1
16), v6.2.3 @ 192.168.10.184:6968 (192.168.10.184), v6.2.3 @ 192.168.10.117:6968 (192.168.10.117), v6.2.3 @ 192.168.10.183:6968 (192.168.10.183), v6.
2.3 @ 192.168.10.143:6968 (192.168.10.143)"}
{"type":"response","@timestamp":"2018-06-20T13:49:20Z","tags":[],"pid":15274,"method":"get","statusCode":200,"req":{"url":"/api/saved_objects/?type=sentinl-watcher&per_page=10&search=%22test%22&search_fields=title&fields=title&page=1","method":"get","headers":{"host":"192.168.10.121:5601","connection":"keep-alive","accept":"application/json, text/plain, */*","kbn-version":"6.2.2","user-agent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.87 Safari/537.36","referer":"http://192.168.10.121:5601/app/sentinl","accept-encoding":"gzip, deflate","accept-language":"en-GB,en-US;q=0.9,en;q=0.8"},"remoteAddress":"192.168.0.108","userAgent":"192.168.0.108","referer":"http://192.168.10.121:5601/app/sentinl"},"res":{"statusCode":200,"responseTime":21,"contentLength":9},"message":"GET /api/saved_objects/?type=sentinl-watcher&per_page=10&search=%22test%22&search_fields=title&fields=title&page=1 200 21ms - 9.0B"}
{"type":"log","@timestamp":"2018-06-20T13:49:20Z","tags":["warning"],"pid":15274,"kibanaVersion":"6.2.2","nodes":[{"version":"6.2.3","http":{"publish_address":"192.168.10.184:6968"},"ip":"192.168.10.184"},{"version":"6.2.3","http":{"publish_address":"192.168.10.143:6968"},"ip":"192.168.10.143"},{"version":"6.2.3","http":{"publish_address":"192.168.10.183:6968"},"ip":"192.168.10.183"},{"version":"6.2.3","http":
{"publish_address":"192.168.10.121:6968"},"ip":"192.168.10.121"},{"version":"6.2.3","http":{"publish_address":"192.168.10.117:6968"},"ip":"192.168.10.117"},{"version":"6.2.3","http":{"publish_address":"192.168.10.182:6968"},"ip":"192.168.10.182"},{"version":"6.2.3","http":{"publish_address":"192.168.10.69:6968"},"ip":"192.168.10.69"},{"version":"6.2.3","http":{"publish_address":"192.168.10.116:6968"},"ip":"192.168.10.116"}],"message":"You're running Kibana 6.2.2 with some different versions of Elasticsearch. Update Kibana or Elasticsearch to the same version to prevent compatibility issues: v6.2.3 @ 192.168.10.184:6968 (192.168.10.184), v6.2.3 @ 192.168.10.143:6968 (192.168.10.143), v6.2.3 @ 192.168.10.183:6968 (192.168.10.183), v6.2.3 @ 192.168.10.121:6968 (192.168.10.121), v6.2.3 @ 192.168.10.117:6968 (192.168.10.117), v6.2.3 @ 192.168.10.182:6968 (192.168.10.182), v6.2.3 @ 192.168.10.69:6968 (192.168.10.69), v6.2.3 @ 192.168.10.116:6968 (192.168.10.116)"}

Best
Aashish

sergibondarenko · 2018-06-20T14:35:25Z

I see no useful info in the Kibana log you posted. Do you see the created watcher in the .kibana index?

curl -XGET localhost:9200/.kibana/doc/_search?pretty

You can also specify a field value to narrow search

curl -XGET localhost:9200/.kibana/_search?q=sentinl-watcher.title:telecom30

sergibondarenko · 2018-06-20T14:35:53Z

Do you see any error in your browser dev console when adding a watcher?

aashish051218 · 2018-06-21T08:43:13Z

This is dev Console Logs

""http://192.168.10.121:5601/app/sentinl"},"res":{"statusCode":200,"responseTime":13,"contentLength":9},"message":"GET /api/saved_objects/?type=senti
nl-script&search=condition*&per_page=50&page=1&search_fields=title%5E3&search_fields=description 200 13ms - 9.0B"}
{"type":"response","@timestamp":"2018-06-21T07:22:08Z","tags":[],"pid":15274,"method":"get","statusCode":200,"req":{"url":"/api/saved_objects/?type=
sentinl-script&search=transform*&per_page=50&page=1&search_fields=title%5E3&search_fields=description","method":"get","headers":{"host":"192.168.10.
121:5601","connection":"keep-alive","accept":"application/json, text/plain, /","kbn-version":"6.2.2","user-agent":"Mozilla/5.0 (X11; Linux x86_64)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.170 Safari/537.36 OPR/53.0.2907.99","referer":"http://192.168.10.121:5601/app/sentinl","acc
ept-encoding":"gzip, deflate","accept-language":"en-GB,en-US;q=0.9,en;q=0.8"},"remoteAddress":"192.168.0.108","userAgent":"192.168.0.108","referer":
"http://192.168.10.121:5601/app/sentinl"},"res":{"statusCode":200,"responseTime":8,"contentLength":9},"message":"GET /api/saved_objects/?type=sentin
l-script&search=transform*&per_page=50&page=1&search_fields=title%5E3&search_fields=description 200 8ms - 9.0B"}
Unhandled rejection [export_exception] Exception when closing export bulk :: {"path":"/_xpack/monitoring/_bulk","query":{"system_id":"kibana","syste
m_api_version":"6","interval":"10000ms"},"body":"{"index":{"_type":"kibana_stats"}}\n{"concurrent_connections":1303,"os":{"load":{"1m"
:0.5361328125,"5m":0
.49365234375,"15m":0.341796875},"memory":{"total_in_bytes":25267707904,"free_in_bytes":2928754688,"used_in_bytes":22338953216},"uptime_in_millis":13376074000},"process":{"event_loop_delay":10001.011863708496,"memory":{"heap":{"total_in_bytes":179527680,"used_in_bytes":155893264,"size_limit":1501560832},"resident_set_size_in_bytes":250347520},"uptime_in_millis":64416429},"requests":{"disconnects":0,"total":659,"status_codes":{"200":248,"304":397,"404":13}},"response_times":{"average":4777,"max":4777},"timestamp":"2018-06-21T07:22:05.330Z","kibana":{"uuid":"192694e9-faaf-4114-8ab7-60ae14082c6c","name":"S25","index":".kibana_main","host":"S25","transport_address":"192.168.10.121:5601","version":"6.2.2","snapshot":false,"status":"green"},"usage":{"index":".kibana_main","dashboard":{"total":33},"visualization":{"total":308},"search":{"total":75},"index_pattern":{"total":47},"graph_workspace":{"total":0},"timelion_sheet":{"total":0},"xpack":{"reporting":{"available":true,"enabled":true,"browser_type":"phantom","_all":0,"csv":{"available":true,"total":0},"printable_pdf":{"available":false,"total":0}}}}}\n","statusCode":500,"response":"{"took":2,"errors":true,"error":{"type":"export_exception","reason":"Exception when closing export bulk","caused_by":{"type":"export_exception","reason":"failed to flush export bulks","caused_by":{"type":"export_exception","reason":"failed to flush export bulk [default_local]","caused_by":{"type":"illegal_state_exception","reason":"There are no ingest nodes in this cluster, unable to forward request to an ingest node."}}}}}"}
at respond (/home/user/kibana-6.2.2-linux-x86_64/node_modules/elasticsearch/src/lib/transport.js:295:15)
at checkRespForFailure (/home/user/kibana-6.2.2-linux-x86_64/node_modules/elasticsearch/src/lib/transport.js:254:7)
at HttpConnector. (/home/user/kibana-6.2.2-linux-x86_64/node_modules/elasticsearch/src/lib/connectors/http.js:159:7)
at IncomingMessage.bound (/home/user/kibana-6.2.2-linux-x86_64/node_modules/elasticsearch/node_modules/lodash/dist/lodash.js:729:21)
at emitNone (events.js:91:20)
at IncomingMessage.emit (events.js:185:7)
at endReadableNT (_stream_readable.js:974:12)
at _combinedTickCallback (internal/process/next_tick.js:80:11)"

aashish051218 · 2018-06-21T08:43:53Z

this is output of the Curl Command
curl -XGET localhost:9200/.kibana/_search?q=sentinl-watcher.title:telecom30

"{"took":1,"timed_out":false,"_shards":{"total":1,"successful":1,"skipped":0,"failed":0},"hits":{"total":0,"max_score":null,"hits":[]}}"

lmangani · 2018-06-21T09:08:00Z

There are no ingest nodes in this cluster, unable to forward request to an ingest node.

This seems to be the core issue, and its a cluster one apparently? does this message appear in the Elasticsearch logs too?

aashish051218 · 2018-06-22T04:29:06Z

Hi Loranzo,
We are using Logstash. Is it pre-requisite for Sentinl to use ES ingest node?

aashish051218 · 2018-06-22T07:09:50Z

Hi Sergi,

There is no sentinl watcher in .kibana index. Does it stores the watchers in .kibana? Sentinl 2 used to store watchers in an index called "Watcher"

aashish051218 · 2018-06-22T10:21:01Z

It is working fine for below configuration:
Kibana 6.2.2
ES: 6.2.2
Sentinl: 6.2.2

But not working for:
Kibana 6.2.2
ES: 6.2.3
Sentinl: 6.2.2

sergibondarenko · 2018-06-25T08:39:06Z

There is no sentinl watcher in .kibana index. Does it stores the watchers in .kibana? Sentinl 2 used to store watchers in an index called "Watcher"

Yes, Sentinl currently stores only in .kibana index. Here are the details #408

@aashish051218 please show me you kibana.yml, sentinl config part.

aashish051218 · 2018-06-26T04:51:10Z

Hi Sergii,
Here are the files.
Kibana.yml.txt
Sentinl json.txt

aashish051218 · 2018-06-28T09:51:57Z

Hi Sergii,
Did you see an anomaly?

Best
Aashish

ss736 · 2019-11-07T09:21:09Z

Hi @aashish051218 ,

If your watcher is getting created successfully but still not able to see in the watcher list in sentinl, then you need to increase the "results" in kibana.yml.

Let me know if you still face the same issue.

Wangyegithubv · 2019-12-26T01:59:26Z

I also face this problem.Can not see the setted watcher lists in sentinl.But the watcher job work normal.
kibana sentinl error:
Sentinl Alarms: An HTTP request has failed to connect. Please check if the Kibana server is running and that your browser has a working connection, or contact your system administrator.

ss736 · 2019-12-28T10:24:00Z

Hi Sergii,
Here are the files.
Kibana.yml.txt
Sentinl json.txt

Hi @aashish051218 ,
I see that there is no results field set in your kibana.yml file. Please set the results field in sentinl section of your kibana.yml and then restart kibana.

Note: set the results values around 1000, if you believe that there will be almost 1000 sentinl alerts.

Reference: https://sentinl.readthedocs.io/en/latest/Config-Example/

The results value show how many sentinl alerts who want to see in Sentinl dashboard.

Let me know if you are still unable to see the alert.

Wangyegithubv · 2019-12-28T23:50:41Z

Hi Sergii,
Here are the files.
Kibana.yml.txt
Sentinl json.txt

Hi @aashish051218 ,
I see that there is no results field set in your kibana.yml file. Please set the results field in sentinl section of your kibana.yml and then restart kibana.

Note: set the results values around 1000, if you believe that there will be almost 1000 sentinl alerts.

Reference: https://sentinl.readthedocs.io/en/latest/Config-Example/

The results value show how many sentinl alerts who want to see in Sentinl dashboard.

Let me know if you are still unable to see the alert.

I set parameters follows:but unable to see the sentinl watcher

job(but the jobs work nomal to alert mails.)

sentinl:
settings:
email:
active: true
host: 10.42..
ssl: false
report:
active: true
tmp_path: /tmp/
and i set follows unable to see job lists too,:

sentinl:
settings:
email:
active: true
host: 10.42.222.51
ssl: false
timeout: 10000 # mail server connection timeout

ss736 · 2019-12-29T13:55:32Z

Hi @aashish051218 ,

You are not following my instruction. Please refer the last comment from me. You will understand the issue. Also, you can refer the attached snapshot for your reference. Make sure you restart Kibana after making the changes in kibana.ym

Wangyegithubv · 2019-12-30T01:50:39Z

Hi @aashish051218 ,

You are not following my instruction. Please refer the last comment from me. You will understand the issue. Also, you can refer the attached snapshot for your reference. Make sure you restart Kibana after making the changes in kibana.ym

Hello,
I set parameters follow:

sentinl:
es:
host: localhost
port: 9200
timefield: '@timestamp'
default_index: watcher
type: sentinl-watcher
alarm_index: watcher_alarms
alarm_type: sentinl-alarm
script_type: sentinl-script
sentinl:
history: 20
results: 50
scriptResults: 50
settings:
email:
active: false
user: ***
password: ***
host: wksowa.wistron.com
ssl: true
timeout: 10000 # mail server connection timeout
But : unable to see the setted alert job lists in sentinl&I also can not set new alert job.

had the error:(the kibana server work ok.and the old sentinl job work nomal too)
Sentinl Watchers: An HTTP request has failed to connect. Please check if the Kibana server is running and that your browser has a working connection, or contact your system administrator.

ss736 · 2019-12-30T03:57:08Z

Hi @aashish051218 ,

Apology for the confusion.

I just wanted you to add "results" field in your kibana.yml. (Do not add any other things in your kibana file). Kindly revert it and just add "results" field in the existing kibana file which was working fine previously(like below)

#Sentinl configuration sentinl: settings: email: active: true user: email@orkash.com password: xxxx host: secure.emailsrvr.com ssl: true report: active: true executable_path: '/usr/bin/google-chrome' sentinl: history: 100 results: 500

Make sure you restart kibana after the change.

Wangyegithubv · 2019-12-30T11:36:29Z

Hi @aashish051218 ,

Apology for the confusion.

I just wanted you to add "results" field in your kibana.yml. (Do not add any other things in your kibana file). Kindly revert it and just add "results" field in the existing kibana file which was working fine previously(like below)

#Sentinl configuration sentinl: settings: email: active: true user: email@orkash.com password: xxxx host: secure.emailsrvr.com ssl: true report: active: true executable_path: '/usr/bin/google-chrome' sentinl: history: 100 results: 500

Make sure you restart kibana after the change.
Version: 5.6.4(kibana&sentinl)
i had setted this:
sentinl:
sentinl:
history: 100
results: 500
settings:
email:
active: true
host: wksowa.wistron.com
ssl: true
timeout: 10000 # mail server connection timeout
report:
active: true
tmp_path: /tmp/
Still unable see job lists.(the elk system can still see it when it's just built On April 2018 but can't see it after a few months,so strange...)

Fix readme. Sync commit from public branch

Wangyegithubv mentioned this issue Dec 26, 2019

I also face this problem.Can not see the setted watcher lists in sentinl.But the watcher job work normal. #762

Closed

lmangani closed this as completed Jul 30, 2020

pull bot pushed a commit to rpatil524/sentinl that referenced this issue Feb 7, 2021

Merge pull request sentinl#469 from sirensolutions/update-readme

a220359

Fix readme. Sync commit from public branch

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Unable to see Watchers in the Sentinl Watcher Table #469

Unable to see Watchers in the Sentinl Watcher Table #469

aashish051218 commented Jun 20, 2018

lmangani commented Jun 20, 2018

aashish051218 commented Jun 20, 2018 •

edited by sergibondarenko

Loading

sergibondarenko commented Jun 20, 2018

sergibondarenko commented Jun 20, 2018 •

edited

Loading

aashish051218 commented Jun 21, 2018

aashish051218 commented Jun 21, 2018

lmangani commented Jun 21, 2018

aashish051218 commented Jun 22, 2018 •

edited

Loading

aashish051218 commented Jun 22, 2018

aashish051218 commented Jun 22, 2018

sergibondarenko commented Jun 25, 2018 •

edited

Loading

aashish051218 commented Jun 26, 2018

aashish051218 commented Jun 28, 2018

ss736 commented Nov 7, 2019 •

edited

Loading

Wangyegithubv commented Dec 26, 2019

ss736 commented Dec 28, 2019

Wangyegithubv commented Dec 28, 2019 •

edited

Loading

ss736 commented Dec 29, 2019

Wangyegithubv commented Dec 30, 2019

ss736 commented Dec 30, 2019 •

edited

Loading

Wangyegithubv commented Dec 30, 2019 •

edited

Loading

Unable to see Watchers in the Sentinl Watcher Table #469

Unable to see Watchers in the Sentinl Watcher Table #469

Comments

aashish051218 commented Jun 20, 2018

lmangani commented Jun 20, 2018

aashish051218 commented Jun 20, 2018 • edited by sergibondarenko Loading

sergibondarenko commented Jun 20, 2018

sergibondarenko commented Jun 20, 2018 • edited Loading

aashish051218 commented Jun 21, 2018

aashish051218 commented Jun 21, 2018

lmangani commented Jun 21, 2018

aashish051218 commented Jun 22, 2018 • edited Loading

aashish051218 commented Jun 22, 2018

aashish051218 commented Jun 22, 2018

sergibondarenko commented Jun 25, 2018 • edited Loading

aashish051218 commented Jun 26, 2018

aashish051218 commented Jun 28, 2018

ss736 commented Nov 7, 2019 • edited Loading

Wangyegithubv commented Dec 26, 2019

ss736 commented Dec 28, 2019

Wangyegithubv commented Dec 28, 2019 • edited Loading

ss736 commented Dec 29, 2019

Wangyegithubv commented Dec 30, 2019

ss736 commented Dec 30, 2019 • edited Loading

Wangyegithubv commented Dec 30, 2019 • edited Loading

aashish051218 commented Jun 20, 2018 •

edited by sergibondarenko

Loading

sergibondarenko commented Jun 20, 2018 •

edited

Loading

aashish051218 commented Jun 22, 2018 •

edited

Loading

sergibondarenko commented Jun 25, 2018 •

edited

Loading

ss736 commented Nov 7, 2019 •

edited

Loading

Wangyegithubv commented Dec 28, 2019 •

edited

Loading

ss736 commented Dec 30, 2019 •

edited

Loading

Wangyegithubv commented Dec 30, 2019 •

edited

Loading