Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Already on GitHub? Sign in to your account

[dev.icinga.com #905] add a command to disable notifications program-wide with expire time as scheduled event; classic ui & idoutils reflected #415

Closed
icinga-migration opened this Issue Oct 15, 2010 · 18 comments

Comments

Projects
None yet
1 participant
Member

icinga-migration commented Oct 15, 2010

This issue has been migrated from Redmine: https://dev.icinga.com/issues/905

Created by rbrinkmo on 2010-10-15 13:46:19 +00:00

Assignee: mfriedrich
Status: Resolved (closed on 2012-08-30 14:51:28 +00:00)
Target Version: 1.8
Last Update: 2012-08-30 14:51:28 +00:00 (in Redmine)


.

Attachments

Changesets

2012-07-30 20:19:26 +00:00 by mfriedrich 5b4e773

core: add a command to disable notifications program-wide with expire time as scheduled event; add commands to cmd.cgi, idoutils programstatus table #905

this adds a new command

DISABLE_NOTIFICATIONS_EXPIRE_TIME;<schedule_time>;<expire_time>

which schedules a new event for expiring disabled notifications ==
re-enable all notifications program-wide.
you can override that expire with ENABLE_NOTIFICATIONS command, not
having an expiry after that. the event for expiry will ignore future
timestamps > event->run_time, so the expiration only takes place on the
correct timestamp.

extinfo.cgi Process Info view adds a new row with "Disable Notifications
Expire Time", showing a readable date time format when notifications are
disabled, and expire time was set.

for cgi.cfg, the default duration can be set as well (default is 1 day).

default_expiring_disabled_notifications_duration=86400

on the event broker side of life, this will be made available within the
programstatus data and updates (which normally happen quite often).
idomod and ido2db have that added to their data structs as well, the
rdbms will get a new column in the programstatus table (cut due to
30char limit of oracle):

disable_notif_expire_time

if one wants to set a start_time, this is currently not
possible/implemented.

other than that, extensive testing is required.

refs #905

2012-08-25 13:22:26 +00:00 by mfriedrich 54b8eff

fix tests, broken by previous commits

refs #905
refs #2878
refs #2725

2012-08-25 23:47:53 +00:00 by mfriedrich 4468b25

idoutils: fix faulty timestamp from #905 on oracle

refs #905

Relations:

Member

icinga-migration commented Nov 5, 2010

Updated by ricardo on 2010-11-05 10:48:32 +00:00

I would like to see that in Icinga.

And implementation is pretty similar to Acknowledges with expire time, execpt that we add a new Command + new timed event.

Member

icinga-migration commented Nov 5, 2010

Updated by mfriedrich on 2010-11-05 11:27:17 +00:00

it still needs a change in the NEB API, and as proposed in #770 our own IEB putting data then into IDO or similar.

if this feature should be implemented, imho an IEB is a must before. thoughts?

Member

icinga-migration commented Nov 5, 2010

Updated by ricardo on 2010-11-05 11:36:20 +00:00

I don't think that this will break the NEB as well.

cause it will be 2 seperated commands.

The ones witch have comments allready, stay as they are. The one's with new, forced comments get submitted as usual and an additional comment command (sounds funny) will be submitted on top of it.

Member

icinga-migration commented Mar 24, 2011

Updated by mfriedrich on 2011-03-24 21:30:43 +00:00

  • Status changed from New to Assigned
  • Assigned to set to ricardo
  • Target Version set to 1.5

i would consider it for icinga 1.5 or 1.6

Member

icinga-migration commented Jun 26, 2011

Updated by mfriedrich on 2011-06-26 15:56:16 +00:00

the question which comes to mind is - this should only be a timely limited command replacement for disable notifications on hosts/services, but by adding another end_time then.

wouldn't it be more sufficient to schedule a downtime, supressing the notifications either way? having set the duration (flexible) and start/endtime (fixed) this would allow basically the same goal to be reached, don't you think?

Member

icinga-migration commented Jul 12, 2011

Updated by mfriedrich on 2011-07-12 18:48:02 +00:00

  • Target Version changed from 1.5 to 1.6
Member

icinga-migration commented Jul 26, 2011

Updated by rbrinkmo on 2011-07-26 15:38:05 +00:00

A scheduled downtime has a different meaning in our SLA reporting. It is like a maintenance (the host/service don't have to be available). That's the reason why we need a different function just to disable the notification for a defined time for those hosts and services which should be available from the view of SLA Reporting.

It would be great if this information also would be available in the idodb.

Member

icinga-migration commented Sep 22, 2011

Updated by mfriedrich on 2011-09-22 18:27:54 +00:00

do you think it would be possible for you to implement that as well next to #770 @ricardo ? if so i would take care about the idoutils part and the docs / web thingies as well.

Member

icinga-migration commented Oct 25, 2011

Updated by mfriedrich on 2011-10-25 09:06:58 +00:00

  • Target Version changed from 1.6 to 1.7

scheduled for later on.

Member

icinga-migration commented Jan 27, 2012

Updated by mfriedrich on 2012-01-27 20:39:44 +00:00

  • Status changed from Assigned to Feedback
  • Target Version changed from 1.7 to 1.8
Member

icinga-migration commented Jul 30, 2012

Updated by mfriedrich on 2012-07-30 15:53:22 +00:00

  • Subject changed from adding a command and function for timely limited deactivation of Notifications to add a command to disable notifications with expire time as scheduled event; classic ui & idoutils reflected
  • Status changed from Feedback to Assigned
  • Assigned to changed from ricardo to mfriedrich

i don't think this should be done on a host or service basis, but globally for all notifications, as we can re-use inner core functionality on this.

  • enable_notifications as variable indicator
  • enable_all_notifications() as reset when the time expired
  • disable_all_notifications() when the command happens, plus scheduling a new expire event

this requires the following changes

core

  • new event: EVENT_EXPIRE_DISABLED_NOTIFICATIONS
  • new command: DISABLE_NOTIFICATIONS_EXPIRE_TIME;<scheduled_time>;<end_time>
    (scheduled_time is ignored, but must be set as DISABLE_NOTIFICATIONS requires this as well!)
  • log a message when the disabled notifications will expire, inkl readable timestamp
  • log a message on expire event, that notifications are now re-enabled
  • retention.dat writes disable_notifications_expire_time
  • retention.dat reads disable_notifications_expire_time, and schedules a new event, if the time did not yet expire (same as ack with expiry time), and notifications are still disabled
  • status.dat reads/writes disable_notifications_expire_time
  • disable_notifications_expire_time is classified as program_status data

cgis

  • disable_notifications_expire_time will be read from status.dat
  • on DISABLE_NOTIFICATIONS command form, the expire time can be ticked, and js shows the expire time textbox to be filled. it then sends a different command, namely DISABLE_NOTIFICATIONS_EXPIRE_TIME.
  • the default cgi.cfg option is default_expiring_disabled_notifications_duration=86400

idoutils

  • idomod passes the variable to ido2db, as protoapi type: IDO_DATA_DISABLED_NOTIFICATIONS_EXPIRE_TIME (also IDO_MAX_DATA_TYPES +1)

  • ido2db will write that to the program_status table, expanding the data fields

  • short name (oracle's 30 char limit) for all: disable_notif_expire_time

    mysql> select * from icinga_programstatus;
    +------------------+-------------+---------------------+---------------------+---------------------+----------------------+------------+-------------+---------------------+---------------------+-----------------------+-------------------------------+--------------------------------+----------------------------+-----------------------------+------------------------+------------------------+----------------------------+--------------------------+-------------------+----------------------+--------------------------+-----------------------------+---------------------------+------------------------------+-----------------------------------+---------------------------+
    | programstatus_id | instance_id | status_update_time | program_start_time | program_end_time | is_currently_running | process_id | daemon_mode | last_command_check | last_log_rotation | notifications_enabled | active_service_checks_enabled | passive_service_checks_enabled | active_host_checks_enabled | passive_host_checks_enabled | event_handlers_enabled | flap_detection_enabled | failure_prediction_enabled | process_performance_data | obsess_over_hosts | obsess_over_services | modified_host_attributes | modified_service_attributes | global_host_event_handler | global_service_event_handler | disable_notifications_expire_time | disable_notif_expire_time |
    +------------------+-------------+---------------------+---------------------+---------------------+----------------------+------------+-------------+---------------------+---------------------+-----------------------+-------------------------------+--------------------------------+----------------------------+-----------------------------+------------------------+------------------------+----------------------------+--------------------------+-------------------+----------------------+--------------------------+-----------------------------+---------------------------+------------------------------+-----------------------------------+---------------------------+
    | 1 | 1 | 2012-07-30 17:51:11 | 2012-07-30 17:47:52 | 0000-00-00 00:00:00 | 1 | 26310 | 1 | 2012-07-30 17:51:11 | 0000-00-00 00:00:00 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 0 | 0 | 1 | 1 | | | 0000-00-00 00:00:00 | 2012-07-30 16:45:09 |
    +------------------+-------------+---------------------+---------------------+---------------------+----------------------+------------+-------------+---------------------+---------------------+-----------------------+-------------------------------+--------------------------------+----------------------------+-----------------------------+------------------------+------------------------+----------------------------+--------------------------+-------------------+----------------------+--------------------------+-----------------------------+---------------------------

what's not possible

  • set a start_time/end_time window. such a command must be called now() lasting til end_time (one could still schedule a cronjob pushing that command at night where maintenance happens)

  • add a comment to that command won't be necessary. the core will log start/stop event to syslog, as well as the disable_notifications_expire_time will be available on status.dat / idoutils for gui selections. to gather an insight who did it, enable cmd.cgi logging in the first place.

    tail -f /var/log/messages | grep -i Disable

    Jul 30 16:40:20 icinga-dev icinga: EXTERNAL COMMAND: DISABLE_NOTIFICATIONS_EXPIRE_TIME;1343659220;1343659509
    Jul 30 16:40:20 icinga-dev icinga: Disabled Notifications will expire on Mon Jul 30 16:45:09 2012 (1343659509)
    Jul 30 16:45:10 icinga-dev icinga: Disabled Notifications expired. All Notifications re-enabled.

might require more tests and bugfixes for now

Member

icinga-migration commented Jul 30, 2012

Updated by mfriedrich on 2012-07-30 16:32:42 +00:00

  • File added icinga_classicui_905_notifications_disabled_expire_time_command_form.png
  • File added icinga_classicui_905_notifications_disabled_expire_time.png
  • File added icinga_classicui_906_notifications_disabled_expire_time_not_set.png

icinga_classicui_906_notifications_disabled_expire_time_not_set.png

icinga_classicui_905_notifications_disabled_expire_time_command_form.png

icinga_classicui_905_notifications_disabled_expire_time.png

Member

icinga-migration commented Jul 30, 2012

Updated by mfriedrich on 2012-07-30 16:36:25 +00:00

the ENABLE_NOTIFICATIONS command should possibly clear

  • disable_notifications_expire_time
  • the scheduled event by that given time

question remains - how to get the timed_event for that given type and time in order to safely delete it? the event itsself should only happen if the time is not cleared before.

Member

icinga-migration commented Jul 30, 2012

Updated by mfriedrich on 2012-07-30 19:30:22 +00:00

  • the expiry will only happen, if disable_notifications_expire_time > 0 (others indicate a forced override by an external ressource)
  • ENABLE_NOTIFICATIONS will case a forced expiry, setting disable_notifications_expire_time = 0
  • if the normal expiry happens, the call to enable_all_notifications must have disable_notifications_expire_time=0 set already to prevent further logging

there might be a race condition on

  • disabling notifications, expire time, schedule expire event
  • forced expire by enable notifications
  • leave scheduled event in the queue
  • disable again with expire time, setting a new expire time

this can be resolved on checking if the expire_time > event_time->run_time, only letting valid expiry times but not cleaning those in the future.

see the test below, which shows 2 events, but only the last one actually happens.

Jul 30 21:10:46 icinga-dev icinga: EXTERNAL COMMAND: DISABLE_NOTIFICATIONS_EXPIRE_TIME;1343675446;1343676025
Jul 30 21:10:46 icinga-dev icinga: Disabled Notifications will expire on Mon Jul 30 21:20:25 2012 (1343676025).
Jul 30 21:10:56 icinga-dev icinga: Disabled Notifications forced expire, re-enabled notifications.
Jul 30 21:11:22 icinga-dev icinga: EXTERNAL COMMAND: DISABLE_NOTIFICATIONS_EXPIRE_TIME;1343675482;1343676305
Jul 30 21:11:22 icinga-dev icinga: Disabled Notifications will expire on Mon Jul 30 21:25:05 2012 (1343676305).
Jul 30 21:25:06 icinga-dev icinga: Disabled Notifications expired. All Notifications re-enabled.

showing that the single one works as well with those adaptions.

Jul 30 21:26:31 icinga-dev icinga: EXTERNAL COMMAND: DISABLE_NOTIFICATIONS_EXPIRE_TIME;1343676391;1343676496
Jul 30 21:26:31 icinga-dev icinga: Disabled Notifications will expire on Mon Jul 30 21:28:16 2012 (1343676496).
Jul 30 21:28:17 icinga-dev icinga: Disabled Notifications expired. All Notifications re-enabled.
Member

icinga-migration commented Jul 30, 2012

Updated by mfriedrich on 2012-07-30 19:54:03 +00:00

  • Subject changed from add a command to disable notifications with expire time as scheduled event; classic ui & idoutils reflected to add a command to disable notifications program-wide with expire time as scheduled event; classic ui & idoutils reflected
Member

icinga-migration commented Jul 30, 2012

Updated by mfriedrich on 2012-07-30 19:59:11 +00:00

hm, i see this was not the original feature request. misread that. follow the host/service notification expiry and discussion on #2919 (with #2473 and #2474 i do not think that this will be necessary, if you can see that someone disabled notifications on a host or service, even a reset button is there already).

Member

icinga-migration commented Aug 1, 2012

Updated by mfriedrich on 2012-08-01 02:33:56 +00:00

  • Status changed from Assigned to Feedback
  • Done % changed from 0 to 90
Member

icinga-migration commented Aug 30, 2012

Updated by mfriedrich on 2012-08-30 14:51:28 +00:00

  • Status changed from Feedback to Resolved
  • Done % changed from 90 to 100

@icinga-migration icinga-migration added this to the 1.8 milestone Jan 17, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment