Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Send non-state notifications for incident #146

Merged
merged 9 commits into from
May 13, 2024
Merged

Send non-state notifications for incident #146

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
8118aab
Support new event types & add `event#Validate()` method
sukhwinder33445 Apr 18, 2024
92d6153
Event: Rename `Acknowledgement` event type to `AcknowledgementSet`
sukhwinder33445 Jan 8, 2024
171dd15
client: Add missing event handeling
sukhwinder33445 Jan 5, 2024
324894c
Format notification subject and message depending on event type
sukhwinder33445 Jan 10, 2024
fd0a00b
Send notifications for non-state events as well
yhabteab Apr 22, 2024
3c3782e
icinga: Fix `RemoveDowntime` unit tests
yhabteab Apr 23, 2024
6617927
Drop unused flapping args `state, stateType`
yhabteab Apr 23, 2024
24429c7
Don't send duplicate notifications for fixed downtimes
yhabteab Apr 25, 2024
e2269d2
Add `Downtime#WasCancelled()` & adjust unit tests
yhabteab May 13, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion cmd/channel/email/main.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,7 @@ func (ch *Email) SendNotification(req *plugin.NotificationRequest) error {
return enmime.Builder().
ToAddrs(to).
From(ch.SenderName, ch.SenderMail).
Subject(fmt.Sprintf("[#%d] %s %s is %s", req.Incident.Id, req.Event.Type, req.Object.Name, req.Incident.Severity)).
Subject(plugin.FormatSubject(req)).
yhabteab marked this conversation as resolved.
Show resolved Hide resolved
Header("Message-Id", fmt.Sprintf("<%s-%s>", uuid.New().String(), ch.SenderMail)).
Text(msg.Bytes()).
Send(ch)
Expand Down
2 changes: 1 addition & 1 deletion cmd/channel/rocketchat/main.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ func main() {

func (ch *RocketChat) SendNotification(req *plugin.NotificationRequest) error {
var output bytes.Buffer
_, _ = fmt.Fprintf(&output, "[#%d] %s %s is %s\n\n", req.Incident.Id, req.Event.Type, req.Object.Name, req.Incident.Severity)
_, _ = fmt.Fprint(&output, plugin.FormatSubject(req)+"\n\n")

plugin.FormatMessage(&output, req)

Expand Down
47 changes: 44 additions & 3 deletions internal/event/event.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,11 +34,52 @@ type Event struct {
}

const (
TypeState = "state"
TypeAcknowledgement = "acknowledgement"
TypeInternal = "internal"
TypeState = "state"
TypeAcknowledgementSet = "acknowledgement-set"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think there's a quite good chance that Web might be unhappy if acknowledgement changes, at least if that happens without prior announcement. @nilmerg (That's what ends up in the event.type column.)

Also depends on the unanswered question whether we actually want to add acknowledgement-cleared as a notification reason in addition to what Icinga 2 allows, if not, this renaming wouldn't even be needed.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think there's a quite good chance that Web might be unhappy if acknowledgement changes, at least if that happens without prior announcement. @nilmerg (That's what ends up in the event.type column.)

And why is that? Web literally does nothing with that column!

Copy link
Collaborator

@julianbrost julianbrost Apr 26, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I haven't checked the source code, @nilmerg said so in #162 for exactly this column:

There may be more added in the future. We need to make sure that there are no such surprises for Web anymore

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

event.type is indeed not handled right now. There's no need for it as there are only state and acknowledge events and only the latter have no severity which is how they're differentiated. This is only relevant once this change is done and Web is adjusted accordingly.

TypeAcknowledgementCleared = "acknowledgement-cleared"
Comment on lines +38 to +39
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are no notifications for ack clear in Icinga 2 at the moment: https://icinga.com/docs/icinga-2/latest/doc/09-object-types/#notification

For the moment, just something I noticed, not yet sure if it's a good or bad idea to have this. Is it something where it would be useful to be notified? Maybe?

TypeInternal = "internal"
TypeDowntimeRemoved = "downtime-removed"
TypeDowntimeStart = "downtime-start"
TypeDowntimeEnd = "downtime-end"
TypeCustom = "custom"
TypeFlappingStart = "flapping-start"
TypeFlappingEnd = "flapping-end"
)

// Validate validates the current event state.
// Returns an error if it detects a misconfigured field.
func (e *Event) Validate() error {
if len(e.Tags) == 0 {
return fmt.Errorf("invalid event: tags must not be empty")
}

if e.SourceId == 0 {
return fmt.Errorf("invalid event: source ID must not be empty")
}

if e.Severity != SeverityNone && e.Type != TypeState {
return fmt.Errorf("invalid event: if 'severity' is set, 'type' must be set to %q", TypeState)
}

switch e.Type {
case "":
return fmt.Errorf("invalid event: 'type' must not be empty")
case TypeState,
TypeAcknowledgementSet,
TypeAcknowledgementCleared,
TypeInternal,
TypeDowntimeRemoved,
TypeDowntimeStart,
TypeDowntimeEnd,
TypeCustom,
TypeFlappingStart,
TypeFlappingEnd:
return nil
default:
return fmt.Errorf("invalid event: unsupported event type %q", e.Type)
}
}

func (e *Event) String() string {
return fmt.Sprintf("[time=%s type=%q severity=%s]", e.Time, e.Type, e.Severity.String())
}
Expand Down
31 changes: 31 additions & 0 deletions internal/icinga2/api_responses.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -91,6 +91,21 @@ type Downtime struct {
Service string `json:"service_name"`
Author string `json:"author"`
Comment string `json:"comment"`

// RemoveTime is used to indicate whether a downtime was ended automatically or cancelled prematurely by a user.
// It is set to zero time for the former case, otherwise to the timestamp at which time has been cancelled.
RemoveTime UnixFloat `json:"remove_time"`
julianbrost marked this conversation as resolved.
Show resolved Hide resolved

// IsFixed is used to differentiate between fixed and flexible downtimes.
// Fixed downtimes always emits a start and triggered event and cause two notifications being sent
// for the very (same) event. Flexible downtimes, on the other hand, only emits a trigger event, and
// don't produce duplicates for the same event.
IsFixed bool `json:"fixed"`
}

// WasCancelled returns true when the current downtime was cancelled prematurely by a user.
func (d *Downtime) WasCancelled() bool {
return d.RemoveTime.Time().After(time.UnixMilli(0))
}

// HostServiceRuntimeAttributes are common attributes of both Host and Service objects.
Expand Down Expand Up @@ -141,6 +156,7 @@ const (
typeDowntimeRemoved = "DowntimeRemoved"
typeDowntimeStarted = "DowntimeStarted"
typeDowntimeTriggered = "DowntimeTriggered"
typeFlapping = "Flapping"
)

// StateChange represents the Icinga 2 API Event Stream StateChange response for host/service state changes.
Expand Down Expand Up @@ -244,6 +260,19 @@ type DowntimeTriggered struct {
Downtime Downtime `json:"downtime"`
}

// Flapping represents the Icinga 2 API Event Stream Flapping response for flapping host/services.
//
// NOTE:
// - An empty Service field indicates a host being in flapping state.
//
// https://icinga.com/docs/icinga-2/latest/doc/12-icinga2-api/#event-stream-type-flapping
type Flapping struct {
yhabteab marked this conversation as resolved.
Show resolved Hide resolved
Timestamp UnixFloat `json:"timestamp"`
Host string `json:"host"`
Service string `json:"service"`
IsFlapping bool `json:"is_flapping"`
}

// UnmarshalEventStreamResponse unmarshal a JSON response line from the Icinga 2 API Event Stream.
//
// The function expects an Icinga 2 API Event Stream Response in its JSON form and tries to unmarshal it into one of the
Expand Down Expand Up @@ -281,6 +310,8 @@ func UnmarshalEventStreamResponse(bytes []byte) (any, error) {
resp = new(DowntimeStarted)
case typeDowntimeTriggered:
resp = new(DowntimeTriggered)
case typeFlapping:
resp = new(Flapping)
default:
return nil, fmt.Errorf("unsupported type %q", responseType)
}
Expand Down
172 changes: 135 additions & 37 deletions internal/icinga2/api_responses_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -128,9 +128,46 @@ func TestObjectQueriesResult_UnmarshalJSON(t *testing.T) {
Name: "dummy-11!af73f9d9-2ed8-45f8-b541-cce3f3fe0f6c",
Type: "Downtime",
Attrs: Downtime{
Host: "dummy-11",
Author: "icingaadmin",
Comment: "turn down for what",
Host: "dummy-11",
Author: "icingaadmin",
Comment: "turn down for what",
RemoveTime: UnixFloat(time.UnixMilli(0)),
IsFixed: true,
},
},
},
{
// $ curl -k -s -u root:icinga 'https://localhost:5665/v1/objects/downtimes' | jq -c '[.results[] | select(.attrs.fixed == false)][1]'
name: "flexible-downtime-host",
jsonData: `{"attrs":{"__name":"dummy-7!691d508b-c93f-4565-819c-3e46ffef1555","active":true,"author":"icingaadmin","authoritative_zone":"","comment":"Flexible","config_owner":"","config_owner_hash":"","duration":7200,"end_time":1714043658,"entry_time":1714040073.241627,"fixed":false,"ha_mode":0,"host_name":"dummy-7","legacy_id":4,"name":"691d508b-c93f-4565-819c-3e46ffef1555","original_attributes":null,"package":"_api","parent":"","paused":false,"remove_time":0,"scheduled_by":"","service_name":"","source_location":{"first_column":0,"first_line":1,"last_column":69,"last_line":1,"path":"/var/lib/icinga2/api/packages/_api/e5ec468f-6d29-4055-9cd4-495dbbef16e3/conf.d/downtimes/dummy-7!691d508b-c93f-4565-819c-3e46ffef1555.conf"},"start_time":1714040058,"templates":["691d508b-c93f-4565-819c-3e46ffef1555"],"trigger_time":1714040073.241627,"triggered_by":"","triggers":[],"type":"Downtime","version":1714040073.241642,"was_cancelled":false,"zone":"master"},"joins":{},"meta":{},"name":"dummy-7!691d508b-c93f-4565-819c-3e46ffef1555","type":"Downtime"}`,
resp: &ObjectQueriesResult[Downtime]{},
expected: &ObjectQueriesResult[Downtime]{
Name: "dummy-7!691d508b-c93f-4565-819c-3e46ffef1555",
Type: "Downtime",
Attrs: Downtime{
Host: "dummy-7",
Author: "icingaadmin",
Comment: "Flexible",
RemoveTime: UnixFloat(time.UnixMilli(0)),
IsFixed: false,
},
},
},
{
// $ curl -k -s -u root:icinga 'https://localhost:5665/v1/objects/downtimes' | jq -c '[.results[] | select(.attrs.fixed == false)][0]'
name: "flexible-downtime-service",
jsonData: `{"attrs":{"__name":"docker-master!disk /!97078a44-8902-495a-9f2a-c1f6802bc63d","active":true,"author":"icingaadmin","authoritative_zone":"","comment":"Flexible","config_owner":"","config_owner_hash":"","duration":7200,"end_time":1714042731,"entry_time":1714039143.459298,"fixed":false,"ha_mode":0,"host_name":"docker-master","legacy_id":3,"name":"97078a44-8902-495a-9f2a-c1f6802bc63d","original_attributes":null,"package":"_api","parent":"","paused":false,"remove_time":0,"scheduled_by":"","service_name":"disk /","source_location":{"first_column":0,"first_line":1,"last_column":69,"last_line":1,"path":"/var/lib/icinga2/api/packages/_api/e5ec468f-6d29-4055-9cd4-495dbbef16e3/conf.d/downtimes/docker-master!disk %2F!97078a44-8902-495a-9f2a-c1f6802bc63d.conf"},"start_time":1714039131,"templates":["97078a44-8902-495a-9f2a-c1f6802bc63d"],"trigger_time":1714039143.459298,"triggered_by":"","triggers":[],"type":"Downtime","version":1714039143.459324,"was_cancelled":false,"zone":""},"joins":{},"meta":{},"name":"docker-master!disk /!97078a44-8902-495a-9f2a-c1f6802bc63d","type":"Downtime"}`,
resp: &ObjectQueriesResult[Downtime]{},
expected: &ObjectQueriesResult[Downtime]{
Name: "docker-master!disk /!97078a44-8902-495a-9f2a-c1f6802bc63d",
Type: "Downtime",
Attrs: Downtime{
Host: "docker-master",
Service: "disk /",
Author: "icingaadmin",
Comment: "Flexible",
RemoveTime: UnixFloat(time.UnixMilli(0)),
IsFixed: false,
},
},
},
Expand All @@ -143,10 +180,12 @@ func TestObjectQueriesResult_UnmarshalJSON(t *testing.T) {
Name: "docker-master!load!c27b27c2-e0ab-45ff-8b9b-e95f29851eb0",
Type: "Downtime",
Attrs: Downtime{
Host: "docker-master",
Service: "load",
Author: "icingaadmin",
Comment: "Scheduled downtime for backup",
Host: "docker-master",
Service: "load",
Author: "icingaadmin",
Comment: "Scheduled downtime for backup",
RemoveTime: UnixFloat(time.UnixMilli(0)),
IsFixed: true,
},
},
},
Expand All @@ -173,7 +212,7 @@ func TestObjectQueriesResult_UnmarshalJSON(t *testing.T) {
LastStateChange: UnixFloat(time.UnixMicro(1697099900637215)),
DowntimeDepth: 0,
Acknowledgement: AcknowledgementNone,
AcknowledgementLastChange: UnixFloat(time.UnixMicro(0)),
AcknowledgementLastChange: UnixFloat(time.UnixMilli(0)),
},
},
},
Expand Down Expand Up @@ -229,7 +268,7 @@ func TestObjectQueriesResult_UnmarshalJSON(t *testing.T) {
LastStateChange: UnixFloat(time.UnixMicro(1697704135756310)),
DowntimeDepth: 0,
Acknowledgement: AcknowledgementNone,
AcknowledgementLastChange: UnixFloat(time.UnixMicro(0)),
AcknowledgementLastChange: UnixFloat(time.UnixMilli(0)),
},
},
},
Expand Down Expand Up @@ -416,9 +455,11 @@ func TestApiResponseUnmarshal(t *testing.T) {
expected: &DowntimeAdded{
Timestamp: UnixFloat(time.UnixMicro(1697207050511293)),
Downtime: Downtime{
Host: "dummy-157",
Author: "icingaadmin",
Comment: "updates",
Host: "dummy-157",
Author: "icingaadmin",
Comment: "updates",
RemoveTime: UnixFloat(time.UnixMilli(0)),
IsFixed: true,
},
},
},
Expand All @@ -428,10 +469,12 @@ func TestApiResponseUnmarshal(t *testing.T) {
expected: &DowntimeAdded{
Timestamp: UnixFloat(time.UnixMicro(1697207141217425)),
Downtime: Downtime{
Host: "docker-master",
Service: "http",
Author: "icingaadmin",
Comment: "broken until Monday",
Host: "docker-master",
Service: "http",
Author: "icingaadmin",
Comment: "broken until Monday",
RemoveTime: UnixFloat(time.UnixMilli(0)),
IsFixed: true,
},
},
},
Expand All @@ -441,9 +484,11 @@ func TestApiResponseUnmarshal(t *testing.T) {
expected: &DowntimeStarted{
Timestamp: UnixFloat(time.UnixMicro(1697207050511378)),
Downtime: Downtime{
Host: "dummy-157",
Author: "icingaadmin",
Comment: "updates",
Host: "dummy-157",
Author: "icingaadmin",
Comment: "updates",
RemoveTime: UnixFloat(time.UnixMilli(0)),
IsFixed: true,
},
},
},
Expand All @@ -453,10 +498,12 @@ func TestApiResponseUnmarshal(t *testing.T) {
expected: &DowntimeStarted{
Timestamp: UnixFloat(time.UnixMicro(1697207141217507)),
Downtime: Downtime{
Host: "docker-master",
Service: "http",
Author: "icingaadmin",
Comment: "broken until Monday",
Host: "docker-master",
Service: "http",
Author: "icingaadmin",
Comment: "broken until Monday",
RemoveTime: UnixFloat(time.UnixMilli(0)),
IsFixed: true,
},
},
},
Expand All @@ -466,9 +513,25 @@ func TestApiResponseUnmarshal(t *testing.T) {
expected: &DowntimeTriggered{
Timestamp: UnixFloat(time.UnixMicro(1697207050511608)),
Downtime: Downtime{
Host: "dummy-157",
Author: "icingaadmin",
Comment: "updates",
Host: "dummy-157",
Author: "icingaadmin",
Comment: "updates",
RemoveTime: UnixFloat(time.UnixMilli(0)),
IsFixed: true,
},
},
},
{
name: "flexible-downtimetriggered-host",
jsonData: `{"downtime":{"__name":"dummy-7!691d508b-c93f-4565-819c-3e46ffef1555","author":"icingaadmin","authoritative_zone":"","comment":"Flexible","config_owner":"","config_owner_hash":"","duration":7200,"end_time":1714043658,"entry_time":1714040073.241627,"fixed":false,"host_name":"dummy-7","legacy_id":4,"name":"691d508b-c93f-4565-819c-3e46ffef1555","package":"_api","parent":"","remove_time":0,"scheduled_by":"","service_name":"","source_location":{"first_column":0,"first_line":1,"last_column":69,"last_line":1,"path":"/var/lib/icinga2/api/packages/_api/e5ec468f-6d29-4055-9cd4-495dbbef16e3/conf.d/downtimes/dummy-7!691d508b-c93f-4565-819c-3e46ffef1555.conf"},"start_time":1714040058,"templates":["691d508b-c93f-4565-819c-3e46ffef1555"],"trigger_time":0,"triggered_by":"","triggers":[],"type":"Downtime","version":1714040073.241642,"zone":"master"},"timestamp":1714040073.242575,"type":"DowntimeAdded"}`,
expected: &DowntimeTriggered{
Timestamp: UnixFloat(time.UnixMicro(1714040073242575)),
Downtime: Downtime{
Host: "dummy-7",
Author: "icingaadmin",
Comment: "Flexible",
RemoveTime: UnixFloat(time.UnixMilli(0)),
IsFixed: false,
},
},
},
Expand All @@ -478,10 +541,41 @@ func TestApiResponseUnmarshal(t *testing.T) {
expected: &DowntimeTriggered{
Timestamp: UnixFloat(time.UnixMicro(1697207141217726)),
Downtime: Downtime{
Host: "docker-master",
Service: "http",
Author: "icingaadmin",
Comment: "broken until Monday",
Host: "docker-master",
Service: "http",
Author: "icingaadmin",
Comment: "broken until Monday",
RemoveTime: UnixFloat(time.UnixMilli(0)),
IsFixed: true,
},
},
},
{
name: "flexible-downtimetriggered-service",
jsonData: `{"downtime":{"__name":"docker-master!disk /!97078a44-8902-495a-9f2a-c1f6802bc63d","author":"icingaadmin","authoritative_zone":"","comment":"Flexible","config_owner":"","config_owner_hash":"","duration":7200,"end_time":1714042731,"entry_time":1714039143.459298,"fixed":false,"host_name":"docker-master","legacy_id":3,"name":"97078a44-8902-495a-9f2a-c1f6802bc63d","package":"_api","parent":"","remove_time":0,"scheduled_by":"","service_name":"disk /","source_location":{"first_column":0,"first_line":1,"last_column":69,"last_line":1,"path":"/var/lib/icinga2/api/packages/_api/e5ec468f-6d29-4055-9cd4-495dbbef16e3/conf.d/downtimes/docker-master!disk %2F!97078a44-8902-495a-9f2a-c1f6802bc63d.conf"},"start_time":1714039131,"templates":["97078a44-8902-495a-9f2a-c1f6802bc63d"],"trigger_time":1714039143.459298,"triggered_by":"","triggers":[],"type":"Downtime","version":1714039143.459324,"zone":""},"timestamp":1714039143.460918,"type":"DowntimeTriggered"}`,
expected: &DowntimeTriggered{
Timestamp: UnixFloat(time.UnixMicro(1714039143460918)),
Downtime: Downtime{
Host: "docker-master",
Service: "disk /",
Author: "icingaadmin",
Comment: "Flexible",
RemoveTime: UnixFloat(time.UnixMilli(0)),
IsFixed: false,
},
},
},
{
name: "downtimeended-host",
jsonData: `{"downtime":{"__name":"dummy-157!e5d4d4ac-615a-4995-ab8f-09d9cd9503b1","author":"icingaadmin","authoritative_zone":"","comment":"updates","config_owner":"","config_owner_hash":"","duration":0,"end_time":1697210639,"entry_time":1697207050.509957,"fixed":true,"host_name":"dummy-157","legacy_id":3,"name":"e5d4d4ac-615a-4995-ab8f-09d9cd9503b1","package":"_api","parent":"","remove_time":0.0,"scheduled_by":"","service_name":"","source_location":{"first_column":0,"first_line":1,"last_column":69,"last_line":1,"path":"/var/lib/icinga2/api/packages/_api/997346d3-374d-443f-b734-80789fd59b31/conf.d/downtimes/dummy-157!e5d4d4ac-615a-4995-ab8f-09d9cd9503b1.conf"},"start_time":1697207039,"templates":["e5d4d4ac-615a-4995-ab8f-09d9cd9503b1"],"trigger_time":1697207050.509957,"triggered_by":"","triggers":[],"type":"Downtime","version":1697207050.509971,"zone":"master"},"timestamp":1697207096.187866,"type":"DowntimeRemoved"}`,
expected: &DowntimeRemoved{
Timestamp: UnixFloat(time.UnixMicro(1697207096187866)),
Downtime: Downtime{
Host: "dummy-157",
Author: "icingaadmin",
Comment: "updates",
RemoveTime: UnixFloat(time.UnixMilli(0)),
IsFixed: true,
},
},
},
Expand All @@ -491,9 +585,11 @@ func TestApiResponseUnmarshal(t *testing.T) {
expected: &DowntimeRemoved{
Timestamp: UnixFloat(time.UnixMicro(1697207096187866)),
Downtime: Downtime{
Host: "dummy-157",
Author: "icingaadmin",
Comment: "updates",
Host: "dummy-157",
Author: "icingaadmin",
Comment: "updates",
RemoveTime: UnixFloat(time.UnixMicro(1697207096187718)),
IsFixed: true,
},
},
},
Expand All @@ -503,10 +599,12 @@ func TestApiResponseUnmarshal(t *testing.T) {
expected: &DowntimeRemoved{
Timestamp: UnixFloat(time.UnixMicro(1697207144746333)),
Downtime: Downtime{
Host: "docker-master",
Service: "http",
Author: "icingaadmin",
Comment: "broken until Monday",
Host: "docker-master",
Service: "http",
Author: "icingaadmin",
Comment: "broken until Monday",
RemoveTime: UnixFloat(time.UnixMicro(1697207144746117)),
IsFixed: true,
},
},
},
Expand Down
Loading
Loading