Add VictorOps Notifier

[rhazdon]

Hi,

I upgraded the alertmanager for sending alerts to VictorOps.
In my company we are using this feature already so please excuse I didn't open a discussion beforehand.

What do you think about it?

[beorn7]

@brian-brazil @fabxc I guess you are more likely than me to have an opinion on this.

[beorn7]

Version change will happen during release, not upon individual commits.

[rhazdon]

Of course. :)
I will fix that asap.

EDIT: Removed.

[mirthy]

Any more progress with this? What do we need to do here?

[rhazdon]

@mirthy Yes, in 2-3 days. I became a daddy two days ago. :)

[rhazdon]

I rebased my fork and solved the merge conflicts.

[mirthy]

Hi guys, looks like @rhazdon made some more changes, any chance someone could look at this again? Sorry to be a pain. Is there anything I can do to help?

[rhazdon]

Hi Guys,
is there anything I should change or improve? :)

[fabxc]

I was on vacation. Am looking now.

[fabxc]

No line break here.

Also this should be using log.Debugf rather than fmt.Printf.

[rhazdon]

Hi @fabxc, thank you very much. You are right, I fixed that. :)

[fabxc]

@rhazdon please also check the comment on the entity ID above.

[rhazdon]

I did. Testing it currently on prodution.

[fabxc]

Thanks. Can you squash that into one commit and then we should be good for merge.

[fabxc]

Nope, compile error ahead.

[fabxc]

👍

[fuzzyami]

@rhazdon
Thanks for your work on VO integration - its something we'd want to use too.

I'm trying it now - and my alerts have payloads ("StateMessage" in VO terms) that could be somewhat improved. The current format contains (among other things) a space-delimited list of values:

"TooManyConfs1 AMS 3.3.3.3 GER-03 1.1.1.1"

This list originates in the key-value pairs that are part of the alert-data in the AlertManager. It would be awesome if that list contained both the keys and the values from the alert:

alertname:TooManyConfs1 datacetner:AMS external_ip:3.3.3.3 region:GER-03 internal_ip:1.1.1.1

Those key/value pairs provide important context to the alert. Without the key, the readability of the alert is greatly reduced. Perhaps this can be easily fixed?

[brian-brazil]

That's a default for all receivers where the message is considered short, as usually you'll know what the values are from experience and there's things like SMS size limits to worry about. You can override it with notification templates.

[fuzzyami]

Thanks, Brian.

For future readers of this thread, I'm pasting below our current VictorOps receiver config. This is a little crude, but it provides all the info we need (alert name is first, then summary and finally full 'payload'). In the future we might customize the summary text and drop the labels.

receivers:
- name: victorOps-receiver
  victorops_configs:
    - api_key: <our_secrect_api_key>
        routing_key: <our_routing_key>
        message: 'Alert: {{ .CommonLabels.alertname }}. Summary:{{ .CommonAnnotations.summary }}. RawData: {{ .CommonLabels }}'