Skip to content

Commit

Permalink
Send custom notification when node has notifies=no
Browse files Browse the repository at this point in the history 
Prior to this commit, if a poller was setup with the setting notifies=no
and you would send a SEND_CUSTOM_HOST/SVC_NOTIFICATION to the Naemon
query handler, then no notification would be sent. This commit ensure
that the master correctly will send the notification.

Note that in the case we handle a custom notification on a node that
normally isn't responsible for the object, then we do not forward the
external command to any other nodes. This is to ensure that we only send
one notification in case the systems has disagreements about the
settings.

This also fixes an issue in general with notifications that might not be
sent in case of pollers with notifies=no.

This fixes MON-10239

Signed-off-by: Jacob Hansen <jhansen@op5.com>
  • Loading branch information
@jacobbaungard
jacobbaungard committed Oct 11, 2018
1 parent b4542b7 commit d04c950
Show file tree
Hide file tree
Showing 2 changed files with 289 additions and 3 deletions.
252 changes: 252 additions & 0 deletions features/notification_propagation.feature
View file
Original file line number Diff line number Diff line change
Expand Up @@ -75,6 +75,7 @@ Feature: A notification should always be handled by the owning node.
When I send naemon command SEND_CUSTOM_SVC_NOTIFICATION;hostA;PONG;4;testCase;A little comment

Then the_poller received event EXTERNAL_COMMAND
And no service notification was sent


Scenario: Custom host notifications sent to Naemon should be blocked and
Expand All @@ -87,6 +88,7 @@ Feature: A notification should always be handled by the owning node.
When I send naemon command SEND_CUSTOM_HOST_NOTIFICATION;hostA;4;testCase;A little comment

Then the_poller received event EXTERNAL_COMMAND
And no host notification was sent


Scenario: Service notifications have been generated and a peer has handled
Expand Down Expand Up @@ -487,3 +489,253 @@ Feature: A notification should always be handled by the owning node.
When I send naemon command ACKNOWLEDGE_SVC_PROBLEM;hostA;PONG;1;1;1;tester;Ack

Then no service notification was sent


Scenario: Custom host notifications sent to Naemon should be executed
on the master if the poller owns the service but doesn't notify.

Given I start naemon with merlin nodes connected
| type | name | port | hostgroup | notifies |
| poller | the_poller | 4002 | pollergroup | no |

When I send naemon command SEND_CUSTOM_HOST_NOTIFICATION;hostA;4;testCase;A little comment

Then the_poller should not receive EXTERNAL_COMMAND
| command_type | 159 |
| command_string | SEND_CUSTOM_HOST_NOTIFICATION |
And 1 host notification was sent
| parameter | value |
| hostname | hostA |


Scenario: Custom host notifications executed from a poller should be
executed on the master if the poller owns the host but doesn't notify.

Given I have merlin config notifies set to no
And I start naemon with merlin nodes connected
| type | name | port | hostgroup |
| master | the_master | 4002 | ignore |

When I send naemon command SEND_CUSTOM_HOST_NOTIFICATION;hostA;4;testCase;A little comment

Then the_master received event EXTERNAL_COMMAND
| command_type | 159 |
| command_string | SEND_CUSTOM_HOST_NOTIFICATION |
And no host notification was sent


Scenario: Custom host notifications executed from a poller should not be
sent to the master if the poller owns the service and is set to notify

Given I have merlin config notifies set to yes
And I start naemon with merlin nodes connected
| type | name | port | hostgroup |
| master | the_master | 4002 | ignore |

When I send naemon command SEND_CUSTOM_HOST_NOTIFICATION;hostA;4;testCase;A little comment

Then the_master should not receive EXTERNAL_COMMAND
| command_type | 159 |
| command_string | SEND_CUSTOM_HOST_NOTIFICATION |
And 1 host notification was sent
| parameter | value |
| hostname | hostA |


Scenario: Custom host notifications should be sent to peers, if the peer
checks the service

Given I start naemon with merlin nodes connected
| type | name | port | hostgroup |
| peer | the_peer | 4002 | ignore |

When I send naemon command SEND_CUSTOM_HOST_NOTIFICATION;hostA;4;testCase;A little comment

Then the_peer received event EXTERNAL_COMMAND
| command_type | 159 |
| command_string | SEND_CUSTOM_HOST_NOTIFICATION |
And no host notification was sent


Scenario: Custom host notifications should not be sent to peers, if the
this node checks the service. This node should also send a notification

Given I have merlin config notifies set to yes
And I start naemon with merlin nodes connected
| type | name | port | hostgroup |
| peer | the_peer | 4002 | ignore |

When I send naemon command SEND_CUSTOM_HOST_NOTIFICATION;hostB;4;testCase;A little comment

Then the_peer should not receive EXTERNAL_COMMAND
| command_type | 159 |
| command_string | SEND_CUSTOM_HOST_NOTIFICATION |
And 1 host notification was sent
| parameter | value |
| hostname | hostB |


Scenario: Peers should recieve custom host notifcations even if it should
be handled by a poller.

Given I start naemon with merlin nodes connected
| type | name | port | hostgroup |
| peer | the_peer | 4002 | ignore |
| poller | the_poller | 4003 | pollergroup |

When I send naemon command SEND_CUSTOM_HOST_NOTIFICATION;hostA;4;testCase;A little comment

Then the_peer received event EXTERNAL_COMMAND
| command_type | 159 |
| command_string | SEND_CUSTOM_HOST_NOTIFICATION |
And the_poller received event EXTERNAL_COMMAND
| command_type | 159 |
| command_string | SEND_CUSTOM_HOST_NOTIFICATION |
And no service notification was sent


Scenario: Custom service notifications sent to Naemon should be executed
on the master if the poller owns the service but doesn't notify.

Given I start naemon with merlin nodes connected
| type | name | port | hostgroup | notifies |
| poller | the_poller | 4002 | pollergroup | no |

When I send naemon command SEND_CUSTOM_SVC_NOTIFICATION;hostA;PONG;4;testCase;A little comment

Then the_poller should not receive EXTERNAL_COMMAND
| command_type | 160 |
| command_string | SEND_CUSTOM_SVC_NOTIFICATION |
And 1 service notification was sent
| parameter | value |
| hostname | hostA |
| servicedesc | PONG |


Scenario: Custom service notifications executed from a poller should be
executed on the master if the poller owns the service but doesn't notify.

Given I have merlin config notifies set to no
And I start naemon with merlin nodes connected
| type | name | port | hostgroup |
| master | the_master | 4002 | ignore |

When I send naemon command SEND_CUSTOM_SVC_NOTIFICATION;hostA;PONG;4;testCase;A little comment

Then the_master received event EXTERNAL_COMMAND
| command_type | 160 |
| command_string | SEND_CUSTOM_SVC_NOTIFICATION |
And no service notification was sent


Scenario: Custom service notifications executed from a poller should not be
sent to the master if the poller owns the service and is set to notify

Given I have merlin config notifies set to yes
And I start naemon with merlin nodes connected
| type | name | port | hostgroup |
| master | the_master | 4002 | ignore |

When I send naemon command SEND_CUSTOM_SVC_NOTIFICATION;hostA;PONG;4;testCase;A little comment

Then the_master should not receive EXTERNAL_COMMAND
| command_type | 160 |
| command_string | SEND_CUSTOM_SVC_NOTIFICATION |
And 1 service notification was sent
| parameter | value |
| hostname | hostA |
| servicedesc | PONG |


Scenario: Custom service notifications should be sent to peers, if the peer
checks the service

Given I start naemon with merlin nodes connected
| type | name | port | hostgroup |
| peer | the_peer | 4002 | ignore |

When I send naemon command SEND_CUSTOM_SVC_NOTIFICATION;hostA;PONG;4;testCase;A little comment

Then the_peer received event EXTERNAL_COMMAND
| command_type | 160 |
| command_string | SEND_CUSTOM_SVC_NOTIFICATION |


Scenario: Custom service notifications should not be sent to peers, if the
this node checks the service. This node should also send a notification

Given I have merlin config notifies set to yes
And I start naemon with merlin nodes connected
| type | name | port | hostgroup |
| peer | the_peer | 4002 | ignore |

When I send naemon command SEND_CUSTOM_SVC_NOTIFICATION;hostB;PONG;4;testCase;A little comment

Then the_peer should not receive EXTERNAL_COMMAND
| command_type | 160 |
| command_string | SEND_CUSTOM_SVC_NOTIFICATION |
And 1 service notification was sent
| parameter | value |
| hostname | hostB |
| servicedesc | PONG |


Scenario: Peers should recieve custom service notifcations even if it should
be handled by a poller.

Given I start naemon with merlin nodes connected
| type | name | port | hostgroup |
| peer | the_peer | 4002 | ignore |
| poller | the_poller | 4003 | pollergroup |

When I send naemon command SEND_CUSTOM_SVC_NOTIFICATION;hostA;PONG;4;testCase;A little comment

Then the_peer received event EXTERNAL_COMMAND
| command_type | 160 |
| command_string | SEND_CUSTOM_SVC_NOTIFICATION |
And the_poller received event EXTERNAL_COMMAND
| command_type | 160 |
| command_string | SEND_CUSTOM_SVC_NOTIFICATION |


Scenario: If two peered pollers have notifies=no then this node should
handle custom notifications.

Given I start naemon with merlin nodes connected
| type | name | port | hostgroup | notifies |
| poller | poller1 | 4002 | pollergroup | no |
| poller | poller2 | 4003 | pollergroup | no |
| peer | peer | 4004 | ignore | yes |

When I send naemon command SEND_CUSTOM_SVC_NOTIFICATION;hostA;PONG;4;testCase;A little comment

Then poller1 should not receive EXTERNAL_COMMAND
| command_type | 160 |
| command_string | SEND_CUSTOM_SVC_NOTIFICATION |
And poller2 should not receive EXTERNAL_COMMAND
| command_type | 160 |
| command_string | SEND_CUSTOM_SVC_NOTIFICATION |
And 1 service notification was sent
| parameter | value |
| hostname | hostA |
| servicedesc | PONG |


Scenario: If one of two peered pollers have notifies=no then this node should
not send out custom notifications for objects handled on the pollers.

Given I start naemon with merlin nodes connected
| type | name | port | hostgroup | notifies |
| poller | poller1 | 4002 | pollergroup | yes |
| poller | poller2 | 4003 | pollergroup | no |
| peer | peer | 4004 | ignore | yes |

When I send naemon command SEND_CUSTOM_SVC_NOTIFICATION;hostA;PONG;4;testCase;A little comment

Then poller1 received event EXTERNAL_COMMAND
| command_type | 160 |
| command_string | SEND_CUSTOM_SVC_NOTIFICATION |
And poller2 received event EXTERNAL_COMMAND
| command_type | 160 |
| command_string | SEND_CUSTOM_SVC_NOTIFICATION |
And no service notification was sent
40 changes: 37 additions & 3 deletions module/hooks.c
View file
Original file line number Diff line number Diff line change
Expand Up @@ -717,6 +717,18 @@ static int hook_external_command(merlin_event *pkt, void *data)
}

node = pgroup_host_node(this_host->id);
/* If we are handling a custom notification, then we need a few extra
* checks to ensure notifications are sent by the master if the
* responsible node has notifies=no.
*/
if (ds->command_type == CMD_SEND_CUSTOM_HOST_NOTIFICATION &&
((!(node->flags & MERLIN_NODE_NOTIFIES) && node != &ipc && node->type == MODE_POLLER) ||
( node == &ipc && (node->flags & MERLIN_NODE_NOTIFIES)))) {
/* as we handle the notification on this system, we do not send out
* the external command to other nodes
*/
return NEB_OK;
}
if (node != &ipc) {
/* We're not responsible, so block this command here */
cb_result = NEBERROR_CALLBACKCANCEL;
Expand Down Expand Up @@ -766,7 +778,19 @@ static int hook_external_command(merlin_event *pkt, void *data)
}

node = pgroup_service_node(this_service->id);
if (node != &ipc) {
/* If we are handling a custom notification, then we need a few extra
* checks to ensure notifications are sent by the master if the
* responsible node has notifies=no.
*/
if (ds->command_type == CMD_SEND_CUSTOM_SVC_NOTIFICATION &&
((!(node->flags & MERLIN_NODE_NOTIFIES) && node != &ipc && node->type == MODE_POLLER) ||
( node == &ipc && (node->flags & MERLIN_NODE_NOTIFIES)))) {
/* as we handle the notification on this system, we do not send out
* the external command to other nodes
*/
return NEB_OK;
}
else if (node != &ipc) {
/* We're not responsible, so block this command here */
cb_result = NEBERROR_CALLBACKCANCEL;
}
Expand Down Expand Up @@ -830,8 +854,9 @@ static int hook_external_command(merlin_event *pkt, void *data)
return 0;
}

if (!merlin_sender)
if (!merlin_sender) {
pkt->hdr.selection = DEST_PEERS_POLLERS;
}
break;
}

Expand Down Expand Up @@ -1009,8 +1034,17 @@ static neb_cb_result * hook_notification(merlin_event *pkt, void *data)
"Notification will be handled by a poller (%s)",
owning_node->name);
} else {
unsigned int i;
int pollers_can_notify = 0;
/* Poller can't notify, who should handle it instead? */
if (num_peers == 0 || should_run_check(object_id)) {
/* First we check if any of the active pollers can notify */
for (i = 0; i < owning_node->pgroup->active_nodes; i++) {
merlin_node *node = owning_node->pgroup->nodes[i];
if (node->flags & MERLIN_NODE_NOTIFIES) {
pollers_can_notify++;
}
}
if (num_peers == 0 || should_run_check(object_id) || pollers_can_notify == 0) {
notif_stats->sent++;
ldebug("notif: Poller can't notify, notifying in its place");
return neb_cb_result_create(0);
Expand Down

0 comments on commit d04c950

Please sign in to comment.