Join GitHub today
GitHub is home to over 31 million developers working together to host and review code, manage projects, and build software together.Sign up
[dev.icinga.com #10156] Client and server check states do not match #3405
This issue has been migrated from Redmine: https://dev.icinga.com/issues/10156
Created by henti on 2015-09-16 10:31:35 +00:00
We had an outage on our monitoring system and since the recovery the state on two services on a remote client keeps changing state resulting in notifications.
I investigated and found that the server and client do not match in service states. The two checks is a proc check for dbsrv12 and a puppet agent check we wrote. the master is in GMT+2 and the client in GMT. below are extracts from the debug logs for the dbsrv12 proc check.
The output of the check command
Updated by mfriedrich on 2015-09-20 11:24:34 +00:00
The master log greps for sql queries for a specific string (which is not necessarily the same service).
I fail to see how the check plugin execution only would correlate to the service state history. There could be different service checks being run, and confusing the execution log. Make sure to 1) compare service objects and their timestamps 2) check the state change history backend on the master.
Updated by henti on 2015-09-21 06:02:41 +00:00
I compared the objects, but could not find any timestamp data. Could you please direct me in the right direction. output from icinga2 object list below :
The state history on the master corresponded with notifications we were seeing.
Updated by mjbrooks on 2015-10-05 09:30:12 +00:00
Same story with this one, @henti pinged me on IRC, he was wondering if you'd seen his feedback and was concerned. I can't seem to change the status back to open so I'm dropping it back in your lap and leaving it as feedback (sorry... again)
Updated by mfriedrich on 2016-02-24 23:15:10 +00:00
Which in term means that the replay log including the check results was not replayed properly. I haven't seen such issues with the recent 2.4.3 release, please test that.