Skip to content

/ icinga2

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[dev.icinga.com #11590] notification interval = 0 not honoured in HA clusters #4121

Closed
icinga-migration opened this issue Apr 14, 2016 · 8 comments
Closed

[dev.icinga.com #11590] notification interval = 0 not honoured in HA clusters #4121

icinga-migration opened this issue Apr 14, 2016 · 8 comments
Labels
area/notifications bug
Milestone
2.5.0

Comments

@icinga-migration
Copy link
Member

@icinga-migration icinga-migration commented Apr 14, 2016

This issue has been migrated from Redmine: https://dev.icinga.com/issues/11590

Created by mnardin on 2016-04-14 09:13:01 +00:00

Assignee: gbeutner
Status: Closed (closed on 2016-06-15 16:09:31 +00:00)
Target Version: 2.5.0
Last Update: 2016-08-08 11:17:30 +00:00 (in Redmine)

Icinga Version: 2.4.4
Backport?: Not yet backported
Include in Changelog: 1

Hi,
in our setup (2 masters, both with notification features enabled) a notification object with interval = 0 will send out more than one notification:

Object 'backeap01-p!soa-state!mail-to-unarm-prod-server' of type 'Notification':
  % declared in '/var/lib/icinga2/api/packages/director/icingam01-p-1460620904-1/zones.d/ris-global/custom-objects/notifications/mail.conf', lines 38:1-38:57
  * __name = "backeap01-p!soa-state!mail-to-unarm-prod-server"
  * command = "mail-service-notification"
    % = modified in '/var/lib/icinga2/api/packages/director/icingam01-p-1460620904-1/zones.d/ris-global/custom-objects/templates/notification-templates.conf', lines 107:3-107:39
  * command_endpoint = ""
  * host_name = "backeap01-p"
    % = modified in '/var/lib/icinga2/api/packages/director/icingam01-p-1460620904-1/zones.d/ris-global/custom-objects/notifications/mail.conf', lines 38:1-38:57
  * interval = 0
    % = modified in '/var/lib/icinga2/api/packages/director/icingam01-p-1460620904-1/zones.d/ris-global/custom-objects/templates/notification-templates.conf', lines 123:5-123:16
  * name = "mail-to-unarm-prod-server"
  * package = "director"
    % = modified in '/var/lib/icinga2/api/packages/director/icingam01-p-1460620904-1/zones.d/ris-global/custom-objects/notifications/mail.conf', lines 38:1-38:57
  * period = "24x7"
    % = modified in '/var/lib/icinga2/api/packages/director/icingam01-p-1460620904-1/zones.d/ris-global/custom-objects/templates/notification-templates.conf', lines 129:5-129:19
  * service_name = "soa-state"
    % = modified in '/var/lib/icinga2/api/packages/director/icingam01-p-1460620904-1/zones.d/ris-global/custom-objects/notifications/mail.conf', lines 38:1-38:57
  * states = [ 1, 2, 4, 8 ]
    % = modified in '/var/lib/icinga2/api/packages/director/icingam01-p-1460620904-1/zones.d/ris-global/custom-objects/templates/notification-templates.conf', lines 109:3-109:45
  * templates = [ "mail-to-unarm-prod-server", "mail-service-notification", "custom-service-notification-properties" ]
    % = modified in '/var/lib/icinga2/api/packages/director/icingam01-p-1460620904-1/zones.d/ris-global/custom-objects/notifications/mail.conf', lines 38:1-38:57
    % = modified in '/var/lib/icinga2/api/packages/director/icingam01-p-1460620904-1/zones.d/ris-global/custom-objects/templates/notification-templates.conf', lines 106:1-106:49
    % = modified in '/var/lib/icinga2/api/packages/director/icingam01-p-1460620904-1/zones.d/ris-global/custom-objects/templates/notification-templates.conf', lines 115:1-115:62
  * times = null
  * type = "Notification"
  * types = [ 32, 16, 64, 8, 128, 256, 1, 2, 4 ]
    % = modified in '/var/lib/icinga2/api/packages/director/icingam01-p-1460620904-1/zones.d/ris-global/custom-objects/templates/notification-templates.conf', lines 110:3-112:57
  * user_groups = null
  * users = [ "unarm" ]
    % = modified in '/var/lib/icinga2/api/packages/director/icingam01-p-1460620904-1/zones.d/ris-global/custom-objects/notifications/mail.conf', lines 42:3-42:21
  * vars = null
  * zone = "RZ"
    % = modified in '/var/lib/icinga2/api/packages/director/icingam01-p-1460620904-1/zones.d/ris-global/custom-objects/notifications/mail.conf', lines 38:1-38:57

debug.log on master1:

[root@icingam01-p ~]# grep 'backeap01-p!soa-state' /var/log/icinga2/debug.log|egrep -v 'add checkable check history|notice/Dependency|Updating reachability'
[2016-04-14 04:02:29 +0200] debug/DbEvents: add log entry history for 'backeap01-p!soa-state'
[2016-04-14 04:02:29 +0200] debug/DbEvents: add state change history for 'backeap01-p!soa-state'
[2016-04-14 04:02:29 +0200] notice/Checkable: State Change: Checkable backeap01-p!soa-state soft state change from OK to CRITICAL detected.
[2016-04-14 04:05:29 +0200] debug/DbEvents: add log entry history for 'backeap01-p!soa-state'
[2016-04-14 04:06:29 +0200] debug/DbEvents: add log entry history for 'backeap01-p!soa-state'
[2016-04-14 04:07:29 +0200] debug/DbEvents: add log entry history for 'backeap01-p!soa-state'
[2016-04-14 04:07:29 +0200] debug/DbEvents: add state change history for 'backeap01-p!soa-state'
[2016-04-14 04:07:29 +0200] notice/Checkable: State Change: Checkable backeap01-p!soa-state hard state change from CRITICAL to CRITICAL detected.
[2016-04-14 04:07:29 +0200] information/Checkable: Checking for configured notifications for object 'backeap01-p!soa-state'
[2016-04-14 04:07:29 +0200] debug/Checkable: Checkable 'backeap01-p!soa-state' has 1 notification(s).
[2016-04-14 04:07:29 +0200] notice/Notification: Attempting to send notifications for notification object 'backeap01-p!soa-state!mail-to-unarm-prod-server'.
[2016-04-14 04:07:29 +0200] information/Notification: Sending notification 'backeap01-p!soa-state!mail-to-unarm-prod-server' for user 'unarm'
[2016-04-14 04:07:29 +0200] debug/DbEvents: add notification history for 'backeap01-p!soa-state'
[2016-04-14 04:07:29 +0200] debug/DbEvents: add contact notification history for service 'backeap01-p!soa-state' and user 'unarm'.
[2016-04-14 04:07:29 +0200] debug/DbEvents: add log entry history for 'backeap01-p!soa-state'
[2016-04-14 04:07:29 +0200] information/Notification: Completed sending notification 'backeap01-p!soa-state!mail-to-unarm-prod-server' for checkable 'backeap01-p!soa-state'
[2016-04-14 08:57:46 +0200] debug/ServiceDbObject: service dependencies for 'backeap01-p!soa-state'
[2016-04-14 08:57:46 +0200] debug/ServiceDbObject: service contacts: backeap01-p!soa-state
[2016-04-14 08:57:46 +0200] debug/ServiceDbObject: service contactgroups: backeap01-p!soa-state
[2016-04-14 09:02:28 +0200] debug/ServiceDbObject: service dependencies for 'backeap01-p!soa-state'
[2016-04-14 09:02:28 +0200] debug/ServiceDbObject: service contacts: backeap01-p!soa-state
[2016-04-14 09:02:28 +0200] debug/ServiceDbObject: service contactgroups: backeap01-p!soa-state
[2016-04-14 09:09:33 +0200] debug/ServiceDbObject: service dependencies for 'backeap01-p!soa-state'
[2016-04-14 09:09:33 +0200] debug/ServiceDbObject: service contacts: backeap01-p!soa-state
[2016-04-14 09:09:33 +0200] debug/ServiceDbObject: service contactgroups: backeap01-p!soa-state

debug.log on master2:

[root@icingam02-p ~]# grep 'backeap01-p!soa-state' /var/log/icinga2/debug.log|egrep -v 'add checkable check history|notice/Dependency|Updating reachability'
[2016-04-14 04:02:29 +0200] debug/DbEvents: add log entry history for 'backeap01-p!soa-state'
[2016-04-14 04:02:29 +0200] debug/DbEvents: add state change history for 'backeap01-p!soa-state'
[2016-04-14 04:02:29 +0200] notice/Checkable: State Change: Checkable backeap01-p!soa-state soft state change from OK to CRITICAL detected.
[2016-04-14 04:05:29 +0200] debug/DbEvents: add log entry history for 'backeap01-p!soa-state'
[2016-04-14 04:06:29 +0200] debug/DbEvents: add log entry history for 'backeap01-p!soa-state'
[2016-04-14 04:07:29 +0200] debug/DbEvents: add log entry history for 'backeap01-p!soa-state'
[2016-04-14 04:07:29 +0200] debug/DbEvents: add state change history for 'backeap01-p!soa-state'
[2016-04-14 04:07:29 +0200] notice/Checkable: State Change: Checkable backeap01-p!soa-state hard state change from CRITICAL to CRITICAL detected.
[2016-04-14 08:57:55 +0200] debug/ServiceDbObject: service dependencies for 'backeap01-p!soa-state'
[2016-04-14 08:57:55 +0200] debug/ServiceDbObject: service contacts: backeap01-p!soa-state
[2016-04-14 08:57:55 +0200] debug/ServiceDbObject: service contactgroups: backeap01-p!soa-state
[2016-04-14 08:57:58 +0200] notice/NotificationComponent: Attempting to send reminder notification for object 'backeap01-p!soa-state'
[2016-04-14 08:57:58 +0200] notice/Notification: Attempting to send notifications for notification object 'backeap01-p!soa-state!mail-to-unarm-prod-server'.
[2016-04-14 08:57:58 +0200] information/Notification: Sending notification 'backeap01-p!soa-state!mail-to-unarm-prod-server' for user 'unarm'
[2016-04-14 08:57:58 +0200] debug/DbEvents: add notification history for 'backeap01-p!soa-state'
[2016-04-14 08:57:58 +0200] debug/DbEvents: add contact notification history for service 'backeap01-p!soa-state' and user 'unarm'.
[2016-04-14 08:57:58 +0200] debug/DbEvents: add log entry history for 'backeap01-p!soa-state'
[2016-04-14 08:57:58 +0200] information/Notification: Completed sending notification 'backeap01-p!soa-state!mail-to-unarm-prod-server' for checkable 'backeap01-p!soa-state'
[2016-04-14 09:02:36 +0200] debug/ServiceDbObject: service dependencies for 'backeap01-p!soa-state'
[2016-04-14 09:02:36 +0200] debug/ServiceDbObject: service contacts: backeap01-p!soa-state
[2016-04-14 09:02:36 +0200] debug/ServiceDbObject: service contactgroups: backeap01-p!soa-state
[2016-04-14 09:09:41 +0200] debug/ServiceDbObject: service dependencies for 'backeap01-p!soa-state'
[2016-04-14 09:09:41 +0200] debug/ServiceDbObject: service contacts: backeap01-p!soa-state
[2016-04-14 09:09:41 +0200] debug/ServiceDbObject: service contactgroups: backeap01-p!soa-state
[2016-04-14 10:02:07 +0200] debug/ServiceDbObject: service dependencies for 'backeap01-p!soa-state'
[2016-04-14 10:02:07 +0200] debug/ServiceDbObject: service contacts: backeap01-p!soa-state
[2016-04-14 10:02:07 +0200] debug/ServiceDbObject: service contactgroups: backeap01-p!soa-state

I don't understand why this happens even with interval = 0:

[2016-04-14 08:57:58 +0200] notice/NotificationComponent: Attempting to send reminder notification for object 'backeap01-p!soa-state'

Regards
Mirko

Relations:
@icinga-migration
Copy link
Member Author

@icinga-migration icinga-migration commented Apr 14, 2016

Updated by mfriedrich on 2016-04-14 09:24:34 +00:00

  • Relates set to 11562
@icinga-migration
Copy link
Member Author

@icinga-migration icinga-migration commented Apr 14, 2016

Updated by mfriedrich on 2016-04-14 09:25:18 +00:00

  • Category set to Notifications

This is most likely a duplicate of #11562.
@icinga-migration
Copy link
Member Author

@icinga-migration icinga-migration commented May 6, 2016

Updated by mfriedrich on 2016-05-06 13:38:06 +00:00

  • Relates deleted 11562
@icinga-migration
Copy link
Member Author

@icinga-migration icinga-migration commented May 6, 2016

Updated by mfriedrich on 2016-05-06 13:38:36 +00:00

  • Parent Id set to 11311
@icinga-migration
Copy link
Member Author

@icinga-migration icinga-migration commented Jun 7, 2016

Updated by mfriedrich on 2016-06-07 19:14:47 +00:00

  • Subject changed from notification interval = 0 not honored to notification interval = 0 not honoured in HA clusters
  • Target Version set to 2.5.0

Left node feels responsible for sending the notification first, then you'll restart/stop the left node. Then the right node is responsible for the notification object. It does not get the last_problem_notification timestamp synced (#11562) and will therefore fire its first local notification. The user gets 2 notifications and thinks that interval = 0 does not work. Apparently it is the missing cluster exchange of notification messages - each node normally calculates the notification times on their own.
@icinga-migration
Copy link
Member Author

@icinga-migration icinga-migration commented Jun 15, 2016

Updated by mfriedrich on 2016-06-15 16:09:31 +00:00

  • Status changed from New to Closed
  • Assigned to set to gbeutner
  • Done % changed from 0 to 100

Fixed as part of #11562 and 8a714d6
@icinga-migration
Copy link
Member Author

@icinga-migration icinga-migration commented Aug 8, 2016

Updated by mfriedrich on 2016-08-08 11:17:30 +00:00

  • Parent Id deleted 11311
@icinga-migration
Copy link
Member Author

@icinga-migration icinga-migration commented Aug 16, 2016

Updated by mfriedrich on 2016-08-16 12:47:17 +00:00

  • Relates set to 12402
@icinga-migration icinga-migration added this to the 2.5.0 milestone Jan 17, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area/notifications bug
Projects
None yet
Milestone
2.5.0
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
1 participant
@icinga-migration