Add msteams #3324

zhan9san · 2023-04-12T08:18:19Z

Related to #3076

As the new line is important to Markdown format, a nested template definition __text_alert_list_markdown is added.

For simplicity, only text field is used to send message in Message card.

The message format is similar to Discord's.

Card design can be checked in Card Playground with following json text.

{
	"@type": "MessageCard",
	"@context": "https://schema.org/extensions",
	"themeColor": "8C1A1A",
	"title": "[FIRING:1] InstanceDown (v critical core)",
	"text": "\n\nAlerts Firing:\n\nLabels:\n - alertname = InstanceDown\n - env = build-service\n - environment = v\n - instance = https://example.com\n - job = blackbox\n - severity = critical\n - type = core\n\nAnnotations:\n - description = https://example.com of job blackbox has been down for more than 5 minutes.\n - summary = Instance https://example.com down\n\nSource: http://v:9090/graph?g0.expr=probe_success%7Btype%3D%22core%22%7D+%3D%3D+0&g0.tab=1\n"
}

kbudde · 2023-04-16T09:50:10Z

docs/configuration.md

+
+```yaml
+# Whether to notify about resolved alerts.
+[ send_resolved: <boolean> | default = true ]


The default is false

Suggested change

[ send_resolved: <boolean> | default = true ]

[ send_resolved: <boolean> | default = false ]

@kbudde

Thanks for your review. I think we need to keep the default value to true.

I have updated the default value in code.

kbudde · 2023-04-16T10:39:28Z

Hey @zhan9san,
LGTM, just change the default in the docs (or change the default in code).

zhan9san · 2023-04-28T06:16:26Z

@simonpasquier

Could you help review this PR?

gotjosh

LGTM!

Thank you so much the contribution 🙏 our users will massively appreciate that. I have left a few nits and some comments please let me know what you think.

gotjosh · 2023-04-28T12:46:25Z

cmd/alertmanager/main.go

@@ -56,6 +56,7 @@ import (
 	"github.com/prometheus/alertmanager/notify/pushover"
 	"github.com/prometheus/alertmanager/notify/slack"
 	"github.com/prometheus/alertmanager/notify/sns"
+	"github.com/prometheus/alertmanager/notify/teams"


nit: Teams is a very common "name" for these type of integrations e.g. Webex also calls their platform Webex Teams.

How do we feel about renaming the package and structs from "teams" to "msteams", I think this is clearer and leaves no room for interpretation.

The package is renamed to msteams in new commit

gotjosh · 2023-04-28T12:48:33Z

docs/configuration.md

@@ -1044,6 +1046,27 @@ attributes:
 [ role_arn: <string> ]
 ```

+### `<teams_config>`
+
+Teams notifications are sent via the [Incoming Webhooks](https://learn.microsoft.com/en-us/microsoftteams/platform/webhooks-and-connectors/what-are-webhooks-and-connectors).


Suggested change

Teams notifications are sent via the [Incoming Webhooks](https://learn.microsoft.com/en-us/microsoftteams/platform/webhooks-and-connectors/what-are-webhooks-and-connectors).

Microsoft Teams notifications are sent via the [Incoming Webhooks](https://learn.microsoft.com/en-us/microsoftteams/platform/webhooks-and-connectors/what-are-webhooks-and-connectors) API endpoint.

It is fixed in new commit

gotjosh · 2023-04-28T12:52:31Z

notify/teams/teams.go

+	webhookURL *config.SecretURL
+}
+
+// Message card https://learn.microsoft.com/en-us/outlook/actionable-messages/message-card-reference


Suggested change

// Message card https://learn.microsoft.com/en-us/outlook/actionable-messages/message-card-reference

// Message card reference can be found at https://learn.microsoft.com/en-us/outlook/actionable-messages/message-card-reference.

It is fixed in new commit

gotjosh · 2023-04-28T13:02:59Z

notify/teams/teams.go

+	resp, err := notify.PostJSON(ctx, n.client, n.webhookURL.String(), &payload)
+	if err != nil {
+		return true, notify.RedactURL(err)
+	}


Suggested change

}

}

defer notify.Drain(resp)

Let's make sure we close the connection.

It is fixed in new commit

gotjosh · 2023-04-28T13:04:53Z

notify/teams/teams.go

+
+	shouldRetry, err := n.retrier.Check(resp.StatusCode, resp.Body)
+	if err != nil {
+		return shouldRetry, err


Suggested change

return shouldRetry, err

return shouldRetry, notify.NewErrorWithReason(notify.GetFailureReasonFromStatusCode(resp.StatusCode), err)

Let's make sure we add a reason to it.

The 429 status code is not set in a standard way. Instead, if it hits rate limit, 200 OK is always send with 'Microsoft Teams endpoint returned HTTP error 429' in response content.

So GetFailureReasonFromStatusCode is renamed to GetFailureReason and add a new parameter responseContent string to address this issue

gotjosh · 2023-04-28T13:07:42Z

notify/teams/teams_test.go

+// This is a test URL that has been modified to not be valid.
+var testWebhookURL, _ = url.Parse("https://example.webhook.office.com/webhookb2/xxxxxx/IncomingWebhook/xxx/xxx")
+
+func TestTeamsRetry(t *testing.T) {


Can we please add 1 more test that is similar to what we have in https://github.com/prometheus/alertmanager/pull/3252/files

I think it's important that we test:

The expected format of the JSON on the webhook api

the reason codes

The test is added in new commit

gotjosh · 2023-04-28T13:08:37Z

notify/teams/teams.go

+	ThemeColor string `json:"themeColor"`
+}
+
+func New(c *config.TeamsConfig, t *template.Template, l log.Logger, httpOpts ...commoncfg.HTTPClientOption) (*Notifier, error) {


Suggested change

func New(c *config.TeamsConfig, t *template.Template, l log.Logger, httpOpts ...commoncfg.HTTPClientOption) (*Notifier, error) {

// New returns a new notifier that uses the Microsoft Teams Webhook API.

func New(c *config.TeamsConfig, t *template.Template, l log.Logger, httpOpts ...commoncfg.HTTPClientOption) (*Notifier, error) {

It is fixed in new commit

simonpasquier · 2023-04-28T14:50:22Z

cmd/alertmanager/main.go

@@ -56,6 +56,7 @@ import (
 	"github.com/prometheus/alertmanager/notify/pushover"
 	"github.com/prometheus/alertmanager/notify/slack"
 	"github.com/prometheus/alertmanager/notify/sns"
+	"github.com/prometheus/alertmanager/notify/teams"


simonpasquier · 2023-04-28T14:54:30Z

notify/teams/teams.go

@@ -0,0 +1,130 @@
+// Copyright 2019 Prometheus Team


(nit)

Suggested change

// Copyright 2019 Prometheus Team

// Copyright 2023 Prometheus Team

It is fixed in new commit

simonpasquier · 2023-04-28T15:01:05Z

notify/teams/teams.go

+	if alerts.Status() == model.AlertFiring {
+		color = colorRed
+	}
+	if alerts.Status() == model.AlertResolved {
+		color = colorGreen
+	}


(nit)

Suggested change

if alerts.Status() == model.AlertFiring {

color = colorRed

}

if alerts.Status() == model.AlertResolved {

color = colorGreen

}

switch alerts.Status() {

case model.AlertFiring:

color = colorRed

case alerts.Status():

color = colorGreen

}

It is addressed in new commit

simonpasquier · 2023-04-28T15:01:31Z

notify/teams/teams.go

+
+	level.Debug(n.logger).Log("incident", key)
+
+	alerts := types.Alerts(as...)


(nit) can you move it right before L99?

It is addressed in new commit

zhan9san · 2023-05-04T05:21:18Z

Thanks for your careful review.

I just get back from a five-day vacation and I'll handle it later.

TheMeier · 2023-05-08T07:21:08Z

@zhan9san thanks a lot four your great contribution. Since the MessageCards are marked "legacy" I was wondering if it might be beneficial to explore switching to AdaptiveCards. Of cause only if that's not a lot of effort or causes more delays in this feature being available.

zhan9san · 2023-05-08T09:04:37Z

@TheMeier

Thanks for pointing it out. I thought about the legacy issue at first.

But to make the PR simple, I just make a lift-shift migration to make MSTeams work.

Besides, this can be considered as an improvement in next PR.

zhan9san · 2023-05-09T19:11:37Z

@simonpasquier and @gotjosh

Would you kindly review this PR?

zhan9san · 2023-05-16T05:22:42Z

Thank you again for taking the time to review the changes.
I just wanted to check in and see if there's anything that's unclear or needs further explanation.

Please let me know if there's anything I can do to make the process easier for you.

notify/msteams/msteams.go

Signed-off-by: Jack Zhang <jack4zhang@gmail.com>

simonpasquier

Sorry for the delay! Can you add msteams to the list of integrations here?

alertmanager/notify/notify.go

Lines 283 to 294 in 83304da

    
           for _, integration := range []string{ 
        
           	"email", 
        
           	"pagerduty", 
        
           	"wechat", 
        
           	"pushover", 
        
           	"slack", 
        
           	"opsgenie", 
        
           	"webhook", 
        
           	"victorops", 
        
           	"sns", 
        
           	"telegram", 
        
           } {

Otherwise this is good for me.

docs/configuration.md

Co-authored-by: Simon Pasquier <spasquie@redhat.com> Signed-off-by: Jack <jack4zhang@gmail.com>

Signed-off-by: Jack Zhang <jack4zhang@gmail.com>

zhan9san · 2023-06-02T14:00:57Z

@simonpasquier

Thanks for your review.

Can you add msteams to the list of integrations here?

It's done.

simonpasquier

Thanks! I'll wait a few days before merging in case @gotjosh has more comments :)

gotjosh

LGTM

gotjosh

LGTM

gotjosh · 2023-06-08T15:27:31Z

Thank you very much for your patience and your contributions ❤️

rgarcia89 · 2023-06-08T16:54:44Z

Now all we need is a new release to be build 😁

SergkeiM · 2023-06-14T09:30:54Z

@zhan9san Amazing thanks a lot for this, we had to use Telegram until MS Teams is added, now we can use 1 messenger only :)

@rgarcia89 I hope will be soon, finger crossed

danielfree · 2023-06-21T09:07:46Z

when can we have a new release? last release was like ~6 months ago..

rgarcia89 · 2023-06-21T18:34:54Z

@gotjosh @simonpasquier any idea when we can count with a new release? It looks like there is quite some demand for this PR to become available 🤔

rgarcia89 · 2023-07-25T05:43:32Z

Anyone tried to build an image including the merge by itself? Is there anything I should give a special look? I think this is the only reasonable way to get an updated image in time

roidelapluie · 2023-07-25T06:05:51Z

You can use the :main image.

* Add msteams Signed-off-by: Jack Zhang <jack4zhang@gmail.com> --------- Signed-off-by: Jack Zhang <jack4zhang@gmail.com> Signed-off-by: Jack <jack4zhang@gmail.com>

gotjosh · 2023-08-25T09:01:14Z

Dear all - this as now been released as part of 0.26

SergkeiM · 2023-08-25T10:50:46Z

Hi @gotjosh and @zhan9san Thank you, already changed it in our Ceph Cluster:
🚀

But having issue in Prometheus though: 3724

yeya24 · 2023-09-01T04:52:58Z

config/config.go

@@ -896,6 +907,7 @@ type Receiver struct {
 	SNSConfigs       []*SNSConfig       `yaml:"sns_configs,omitempty" json:"sns_configs,omitempty"`
 	TelegramConfigs  []*TelegramConfig  `yaml:"telegram_configs,omitempty" json:"telegram_configs,omitempty"`
 	WebexConfigs     []*WebexConfig     `yaml:"webex_configs,omitempty" json:"webex_configs,omitempty"`
+	MSTeamsConfigs   []*MSTeamsConfig   `yaml:"msteams_configs,omitempty" json:"teams_configs,omitempty"`


Should we align the configuration name? I am wondering why we call msteams for yaml but teams for json.
@zhan9san @simonpasquier

@yeya24

Thanks for pointing it out. It has to be a typo.

I'll submit a PR to fix it.

Nice catch!

* Add msteams Signed-off-by: Jack Zhang <jack4zhang@gmail.com> --------- Signed-off-by: Jack Zhang <jack4zhang@gmail.com> Signed-off-by: Jack <jack4zhang@gmail.com>

zhan9san force-pushed the feature/add-msteams branch from 1d5b341 to 1136511 Compare April 12, 2023 08:34

kbudde suggested changes Apr 16, 2023

View reviewed changes

kbudde mentioned this pull request Apr 16, 2023

Direct integration into Alertmanager? prometheus-msteams/prometheus-msteams#316

Closed

zhan9san force-pushed the feature/add-msteams branch from 1136511 to eb1f936 Compare April 16, 2023 10:51

kbudde approved these changes Apr 16, 2023

View reviewed changes

simonpasquier self-requested a review April 16, 2023 15:38

gotjosh reviewed Apr 28, 2023

View reviewed changes

gotjosh added the waiting-for-user-input The Pull Request or Issue is waiting for input from the user that reported it or created it. label Apr 28, 2023

simonpasquier reviewed Apr 28, 2023

View reviewed changes

zhan9san force-pushed the feature/add-msteams branch 2 times, most recently from bf3b5ef to 787079b Compare May 9, 2023 18:58

zhan9san requested review from gotjosh and simonpasquier May 16, 2023 08:46

simonpasquier reviewed May 23, 2023

View reviewed changes

notify/msteams/msteams.go Show resolved Hide resolved

zhan9san added 2 commits May 24, 2023 21:36

Add msteams

3d70e93

Signed-off-by: Jack Zhang <jack4zhang@gmail.com>

use msteams as package name and add one more unit test

55cb0d7

Signed-off-by: Jack Zhang <jack4zhang@gmail.com>

zhan9san force-pushed the feature/add-msteams branch from 787079b to 13bda64 Compare May 24, 2023 14:02

Revert the retry handling logic

9e017b6

Signed-off-by: Jack Zhang <jack4zhang@gmail.com>

zhan9san force-pushed the feature/add-msteams branch from 13bda64 to 9e017b6 Compare May 24, 2023 14:04

simonpasquier reviewed Jun 2, 2023

View reviewed changes

docs/configuration.md Outdated Show resolved Hide resolved

zhan9san and others added 2 commits June 2, 2023 21:46

Update docs/configuration.md

df260d7

Co-authored-by: Simon Pasquier <spasquie@redhat.com> Signed-off-by: Jack <jack4zhang@gmail.com>

Add msteams in NewMetrics

da34017

Signed-off-by: Jack Zhang <jack4zhang@gmail.com>

simonpasquier approved these changes Jun 2, 2023

View reviewed changes

gotjosh approved these changes Jun 8, 2023

View reviewed changes

gotjosh merged commit d10f21f into prometheus:main Jun 8, 2023
5 checks passed

zhan9san mentioned this pull request Jul 4, 2023

Define a RELEASE.MD for the Alertmanager Project #3344

Closed

rgarcia89 mentioned this pull request Jul 25, 2023

Add msteams configuration to Alertmanager Config CRD prometheus-operator/prometheus-operator#5766

Closed

ksdpmx mentioned this pull request Jul 30, 2023

feat: add support for MSTeams receiver prometheus-operator/prometheus-operator#5778

Closed

5 tasks

gotjosh pushed a commit that referenced this pull request Aug 7, 2023

Add msteams (#3324)

a8c3cfb

* Add msteams Signed-off-by: Jack Zhang <jack4zhang@gmail.com> --------- Signed-off-by: Jack Zhang <jack4zhang@gmail.com> Signed-off-by: Jack <jack4zhang@gmail.com>

zhan9san mentioned this pull request Aug 18, 2023

fix(helm): changing envFrom type prometheus-msteams/prometheus-msteams#341

Closed

zhan9san deleted the feature/add-msteams branch August 25, 2023 09:11

yeya24 reviewed Sep 1, 2023

View reviewed changes

zhan9san mentioned this pull request Sep 1, 2023

update msteams configs #3495

Merged

luka5 mentioned this pull request Oct 9, 2023

Fix Docs: default for msteams config #3550

Merged

nicolastakashi mentioned this pull request Oct 12, 2023

feat: adding alertmanager msteams support prometheus-operator/prometheus-operator#6002

Merged

5 tasks

rgarcia89 mentioned this pull request Dec 18, 2023

MSTeams integration message cards actions not supported #3642

Open

	[ send_resolved: <boolean> \| default = true ]
	[ send_resolved: <boolean> \| default = false ]

	Teams notifications are sent via the [Incoming Webhooks](https://learn.microsoft.com/en-us/microsoftteams/platform/webhooks-and-connectors/what-are-webhooks-and-connectors).
	Microsoft Teams notifications are sent via the [Incoming Webhooks](https://learn.microsoft.com/en-us/microsoftteams/platform/webhooks-and-connectors/what-are-webhooks-and-connectors) API endpoint.

	// Message card https://learn.microsoft.com/en-us/outlook/actionable-messages/message-card-reference
	// Message card reference can be found at https://learn.microsoft.com/en-us/outlook/actionable-messages/message-card-reference.

	return shouldRetry, err
	return shouldRetry, notify.NewErrorWithReason(notify.GetFailureReasonFromStatusCode(resp.StatusCode), err)

	func New(c config.TeamsConfig, t template.Template, l log.Logger, httpOpts ...commoncfg.HTTPClientOption) (*Notifier, error) {
	// New returns a new notifier that uses the Microsoft Teams Webhook API.
	func New(c config.TeamsConfig, t template.Template, l log.Logger, httpOpts ...commoncfg.HTTPClientOption) (*Notifier, error) {

	// Copyright 2019 Prometheus Team
	// Copyright 2023 Prometheus Team


		level.Debug(n.logger).Log("incident", key)

		alerts := types.Alerts(as...)

	for _, integration := range []string{
	"email",
	"pagerduty",
	"wechat",
	"pushover",
	"slack",
	"opsgenie",
	"webhook",
	"victorops",
	"sns",
	"telegram",
	} {

Add msteams #3324

Add msteams #3324

Conversation

zhan9san commented Apr 12, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kbudde commented Apr 16, 2023

zhan9san commented Apr 28, 2023

gotjosh left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zhan9san commented May 4, 2023

TheMeier commented May 8, 2023 • edited

zhan9san commented May 8, 2023

zhan9san commented May 9, 2023

zhan9san commented May 16, 2023

simonpasquier left a comment

Choose a reason for hiding this comment

zhan9san commented Jun 2, 2023

simonpasquier left a comment

Choose a reason for hiding this comment

gotjosh left a comment

Choose a reason for hiding this comment

gotjosh left a comment

Choose a reason for hiding this comment

gotjosh commented Jun 8, 2023

rgarcia89 commented Jun 8, 2023

SergkeiM commented Jun 14, 2023

danielfree commented Jun 21, 2023

rgarcia89 commented Jun 21, 2023 • edited

rgarcia89 commented Jul 25, 2023

roidelapluie commented Jul 25, 2023

gotjosh commented Aug 25, 2023

SergkeiM commented Aug 25, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zhan9san commented Apr 12, 2023 •

edited

TheMeier commented May 8, 2023 •

edited

rgarcia89 commented Jun 21, 2023 •

edited

SergkeiM commented Aug 25, 2023 •

edited