/var/log/icinga2/icinga2.log is growing very fast on satellites #6542
Comments
|
Awesome, opening an issue for this has been on my agenda, too. Same here: Your Environment
Currently we are "handling" this problem by creating a dedicated check_disk service for the logging volume /var, which executes an EventCommand @ warning free space <15%. The EventCommand calls a shell script which executes: Rotating & zipping the log may be another approach to handle this, depending on the need to keep the logs. |
|
More thoughts on this: The commits are not exclusively triggered by Director's Deployments. I triggered 4 Deploys this morning: The icinga2.log counts much more sets of Apply Rule mismatches (generally headed by the message 'Commiting config item(s)', so i grepped for it): Each commit triggers 690 warnings of Apply Rule mismatches on one of our satellites in Zone A, which seems valid, since theses services need to be applied to hosts in a different Zone B. The log on a sattelite in Zone B looks vice versa: 294 warnings of Apply Rule mismatches, because these services need to be deployed at Zone A. It may also be helpful to know the reason for the huge amount of commits. |
|
@dbu1986 a script for deleting the log file would be a work-around, we thought also in our department. But it first it would be nice if this would be patched and like you wrote why this happend. |
|
When apply rules are configured, and they do not match anywhere, the user should be made aware of them. This is a general principle with Icinga and the config compiler, and has been sitting there since 2.0. I'm not sure which change has had an impact on generating more of such non-matching apply rules. In any case, I would analyse where these apply rules source from, and why they are deployed into the satellite node. Highly likely they do exist as global zone and are synced to any cluster node, or they are wrongly pushed to the satellite via cluster config sync and the corresponding zone. Removing the configuration warning, or changing its verbosity level is not an option here, as this would remove a well known configuration validation feature used in many places. |
dnsmichi
added
question
area/configuration
|
@dnsmichi thanks for your answer. |
|
I'm currently on vacation, just some quick thoughts on this:
Cheers, |
Yes, absolutely! I highly appreciate these messages and it's not my intention to get them suppressed. The Servicesets are created via Director, and deployed via global zone "director-global". Here is an example: This seems to be standard behaviour. Director currently does not offer an option to set the zone per Serviceset (Icinga/icingaweb2-module-director#1589). In my environment the global zone "director-global" is configured on both master nodes and all satellite nodes (2 satellites per zone). Aside of that I have a global zone "globalconfig" configured on all nodes (Masters, Satellites, and client hosts) for deploying Director's CheckCommands. The intention of using these two global zones was to avoid non-matching Apply Rule warnings at all client hosts. However, I'm confident that the logs will get much smaller if we get the deploying of apply rules solved, I think that is a side effect. But I'm not sure if we have another issue here which causes the high rate of commits. Anyway there still would be a questionble high rate of "Committing config item(s)" mentions in the log. We also have a problem of reload looping. Sorry for not mentioning earlier, it just came to my mind. At this moment the icinga2.log's look like this: MasterA (configuration master): MasterB: satellite_z1n1 & satellite_z1n2: Any advice, how I can get this under control? I've been struggling for days trying to debug when this starts and when it ends. Currently I'm establishing another separate environment for testing our config, but for now I don't have any clue where this is coming from. |
|
@Thomas-Gelf thanks for the answer. Maybe it is a mixture of more things? But I think it reduces the warnings in the logs, if the director deploy the servicesets in the correct zones. For example in our case if the serviceset will deploy it only for the server-zone and not for the network Zone and vice versa. One information for you: in our configuration we import the hosts every 2 hours from our CMDB via CSV. If there are changes the directore will sync. Sometimes we have to do some manual configuration. But why icinga produce so much warnings is the other point. For now: enjoy your holidays ;-) you deserve it! You are doing a good job with the director! |
|
@stevie-sy: glad to hear that, thank you! Just wanted to point you to that discrepancy in your numbers, you should try to dig deeper. Eventually you can track down a looping behavior similar to what @dbu1986 describes. Your log file is so huge, take an average log line length and do the math on your own. Regular deployments can hardly account for even 0.1% of that data. |
|
I'm wondering why these messages are coming in from the https://github.com/Icinga/icinga2/blob/master/lib/remote/apilistener-configsync.cpp#L79 Please show the Endpoint and Zone configuration on your master, including all the satellites. |
|
I want to give you a small update to this Issue: Maybe this [(https://github.com//pull/6663]) from the update 2.10.1 helped. If @dbu1986 didn't get this Problem also again, maybe this could be closed. |
|
Likely it helps a bit if you have many created objects and non-matching apply rules. It popped into my mind how to silence these logged lines, so I just did it. |
|
And you did a good job (y). |
|
The thing I'm still curious about is that log line This must never happen, and I want to know how to reproduce this. @dbu1986 Unfortunately the zones configuration doesn't tell me much, how's the configuration deployed and synced top down, what exactly causes this behaviour? |
I assume you are reffering to #L224 ? The log message might be related to the ConfigDeleteObjectAPIHandler, since here no further details on the discarded object are being written into the log (following
I'd desperately love to reveal that, too 😄 Unfortunately I have no idea what triggers it and how to find a point to start from. However, I'm not sure if I got your question well, but as mentioned above, we're using HA distributed monitoring infrastructure in top/down mode, containing 2 Master, 3 Zones with 2 Satellites each, 4 Zones with 1 Satellite each, and a load of clients per Zone. Most of the objects are managed by Director (~10.000 Services, ~800 Hosts), plus ~5.000 Services created as runtime objects via REST-API on the config master, plus <20 Services defined in a configfile on the config master. Additionally we are using a ticket system, fed by Icinga2 Notifications (passed in a perl script via REST API), and as a return path the ticket system sends Acknowledgements and Comments back to Icinga2 asynchronously. One thing coming up my mind is the discrepancy between the timestamps of these messages. Today's messages as an example: satellite_z1n1: I somehow expected the message timestamps to nearly match to each other, but the 'discarding' messages at the master started around 04:06 and the 'sending config' messages at the satellite started much later, around 08:53. So there may be config updates on the satellite, that I am totally missing. Director's deployment log is listing today's first deployment at 07:52. And the icinga2 system processes are up and running since a couple of days, so I can also exclude a restart of Icinga2. A few log lines from satellite_z1n1 before the first 'sending config' message to MasterA: Could the 'API client disconnected' message be related to this? |
|
We have the same problem here. Your Environment
This is flooding our logs.
|
|
Are you syncing the |
|
For further tests, please remove any |
|
@lippserd @dnsmichi After lots of other tasks, tech. problems and (uncached) Icinga 2 builds I've finally set up a playground for this and tried to reproduce the issue. Either I've overseen an elephant (again) or there's something really interesting (and creepy) with that ScheduledDowntimes. @dnsmichi Let's talk offline about the tech. details on monday. |
|
The ScheduledDowntimes by themselves purr like a cat. But the created downtimes disappear suddenly. |
|
The web app shown here polls the Icinga 2 API all the time and appends new downtimes to its list and colors them green. Once a downtime disappear, it colors it red. |
|
This seems to be the following case:
Both calls lead into lib/remote/configobjectutility.cpp where CreateObject() always logs that it has created a runtime object (a downtime in this case). When a downtime is deleted, nothing is logged. Therefore a local loop exists which immediately stops on restart, but then begins again after the expired downtime handler kicks in. It doesn't matter whether the zone is the master or the satellite, creation and deletion happens locally. |
dnsmichi
added
bug
|
Can you please test the snapshot packages, it includes all referenced issues and PRs. Thanks, |
|
ref/NC/585559 |
dnsmichi
removed
the
needs feedback
|
ref/IP/9673 reported this working as expected. Since there's no more feedback here, I consider this being resolved. |
We have a HA configuration with 2 master nodes, 2 satellites in the Network Zone and 2 satellites in the Server Zone. For configuration we use the director.
Since some time - we noticed it after upgrade to Version 2.9.1 - that icinga2.log ist growing and growing very fast from one point to another. But only on the satellites in the Network Zone. It seems it happend after deployment with the director.
For example: that are times when there was a deplolyment today:
That is the visualisation with grafana where you can see the growing of the log file:
and direct in the bash:
I think the problem are the ServiceSets which wie generate with the director (see also Icinga/icingaweb2-module-director#1589). Because it will deploy them at every node. So if the ServerSet is for example for the Server Zone, the satellite in the Network Zone won't find the devices to apply. And if I take a look into the icinga2.log at the time of the depolyment ist is full of Messages like this:
the only work arround is to empty the file with
> icinga2.log
Your Environment
•Director version (System - About): Git Master 71ad855
•Icinga Web 2 version and modules (System - About): 2.6.1
•Icinga 2 version (icinga2 --version): r2.9.1-1
•Operating System and version: CentOS 7
•Webserver, PHP versions: Apache 2.4.6-80.el7, rh-php 7.1.8-1.el7
The text was updated successfully, but these errors were encountered: