-
Notifications
You must be signed in to change notification settings - Fork 572
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Restarting Icinga service causes lots of alerts (systemd) #6873
Comments
That sounds familiar. Could you provide the log output around a restart? It'd be interesting why Icinga 2 kills these checks. |
@sammcj can you please provide that while I'm away? |
I received a report from a colleague who can reproduce this, logs are sadly of no help. We currently suspect systemd to be the culprit, killing child processes before we can terminate them. The solution would be to ignore checkresults when we are restarting/quitting and rescheduling them to be run when icinga2 runs again. |
Possible fix: #6908 |
I'd even say it's the fix as it doesn't let bad check results even happen instead of ignoring and re-scheduling. |
Any update on this? I see #6908 was merged - did that fix it? |
@sammcj Please could you test our snapshot packages and report the result? |
Will do that tomorrow as it's 23:26 in Melbourne, Australia |
@Al2Klimov FYI @alexizmailov works with / near me, he's pointed out that while it's been merged it hasn't been released yet so once it has been released, we'll install - test and update this ticket. Thanks :) |
That's a bit tricky, as we're waiting for your feedback to release this. So we'd appreciate it if you can test the snapshot package in your environment prior to any release :) |
No, unfortunately the problem still persists, tested 3 times:
|
I'm afraid that fix is only the half rent. I just installed a fresh Icinga 2 on a fresh CentOS 7 – our icinga2.service doesn't specify any KillMode. The default is control-group so systemd kills all check plugins. |
The mentioned fix also causes other problems with restart delays and missing Stop() calls then. I will partially revert this and re-evaluate a possible fix for the kill problem. |
@alexizmailov Please try to change |
I will try this tomorrow because it's 22:45 now in Melbourne, |
Looks like it works, I restarted it 4 times and there were no alerts at all. |
Expected Behavior
systemctl restart icinga2
- no alerts after.Current Behavior
After restarting the service lots of alerts appear on the dashboard. All checks that were running will return
critical
.Possible Solution
Do not alert on <terminated by signal 15>
Steps to Reproduce (for bugs)
Screenshot
Context
Your Environment
icinga2 --version
):version: r2.10.2-1
CentOS Linux release 7.6.1810 (Core)
icinga2 feature list
):api checker command compatlog graphite ido-pgsql influxdb livestatus mainlog notification
2.6.2
icinga2 daemon -C
):output
zones.conf
file (oricinga2 object list --type Endpoint
andicinga2 object list --type Zone
) from all affected nodes.output
The text was updated successfully, but these errors were encountered: