level=error msg="Failed sending to webhook url ...: Forbidden"

[revelnode]

Hello, I am trying to test out the functionality of the prometheus-msteam pod by directly sending it a message using your template. I get this in retunr (URL modified for privacy)

/ # curl -X POST -d @prom-alert.json http://prometheus-msteams:2000/alertmanager
Failed sending to webhook url https://outlook.office.com/webhook/xxx/xxx. Got the error: Post https://outlook.office.com/webhook/xxx/xxx: Forbidden

I check the logs, and I see this:

time="2019-12-03T22:40:58Z" level=info msg="/alertmanager received a request"
time="2019-12-03T22:40:59Z" level=debug msg="Prometheus Alert: {\"receiver\":\"teams_proxy\",\"status\":\"firing\",\"alerts\":[{\"status\":\"\",\"labels\":{\"alertname\":\"high_memory_load\",\"instance\":\"10.80.40.11:9100\",\"job\":\"docker_nodes\",\"monitor\":\"master\",\"severity\":\"warning\"},\"annotations\":{\"description\":\"xxxxxxx\",\"summary\":\"Server High Memory usage\"},\"startsAt\":\"2018-03-07T06:33:21.873077559-05:00\",\"endsAt\":\"0001-01-01T00:00:00Z\",\"generatorURL\":\"\"}],\"groupLabels\":{\"alertname\":\"high_memory_load\"},\"commonLabels\":{\"alertname\":\"high_memory_load\",\"monitor\":\"master\",\"severity\":\"warning\"},\"commonAnnotations\":{\"summary\":\"Server High Memory usage\"},\"externalURL\":\"http://alertmanager:9093\",\"version\":\"4\",\"groupKey\":\"{}:{alertname=\\\"high_memory_load\\\"}\"}"
time="2019-12-03T22:40:59Z" level=debug msg="Alert rendered in template file: \n{\n  \"@type\": \"MessageCard\",\n  \"@context\": \"http://schema.org/extensions\",\n  \"themeColor\": \"FFA500\",\n  \"summary\": \"Server High Memory usage\",\n  \"title\": \"Prometheus Alert (firing)\",\n  \"sections\": [ \n    {\n      \"activityTitle\": \"[xxxxxxx](http://alertmanager:9093)\",\n      \"facts\": [\n        {\n          \"name\": \"description\",\n          \"value\": \"xxxxxxx\"\n        },\n        {\n          \"name\": \"summary\",\n          \"value\": \"Server High Memory usage\"\n        },\n        {\n          \"name\": \"alertname\",\n          \"value\": \"high memory load\"\n        },\n        {\n          \"name\": \"instance\",\n          \"value\": \"10.80.40.11:9100\"\n        },\n        {\n          \"name\": \"job\",\n          \"value\": \"docker nodes\"\n        },\n        {\n          \"name\": \"monitor\",\n          \"value\": \"master\"\n        },\n        {\n          \"name\": \"severity\",\n          \"value\": \"warning\"\n        }\n      ],\n      \"markdown\": true\n    }\n  ]\n}\n"
time="2019-12-03T22:40:59Z" level=debug msg="Size of message is 557 Bytes (~0 KB)"
time="2019-12-03T22:40:59Z" level=info msg="Created a card for Microsoft Teams /alertmanager"
time="2019-12-03T22:40:59Z" level=debug msg="Teams message cards: [{\"@type\":\"MessageCard\",\"@context\":\"http://schema.org/extensions\",\"themeColor\":\"FFA500\",\"summary\":\"Server High Memory usage\",\"title\":\"Prometheus Alert (firing)\",\"sections\":[{\"activityTitle\":\"[xxxxxxx](http://alertmanager:9093)\",\"facts\":[{\"name\":\"description\",\"value\":\"xxxxxxx\"},{\"name\":\"summary\",\"value\":\"Server High Memory usage\"},{\"name\":\"alertname\",\"value\":\"high memory load\"},{\"name\":\"instance\",\"value\":\"10.80.40.11:9100\"},{\"name\":\"job\",\"value\":\"docker nodes\"},{\"name\":\"monitor\",\"value\":\"master\"},{\"name\":\"severity\",\"value\":\"warning\"}],\"markdown\":true}]}]"
time="2019-12-03T22:40:59Z" level=error msg="Failed sending to webhook url https://outlook.office.com/webhook/xxx/xxx. Got the error: Post https://outlook.office.com/webhook/xxx/xxx: Forbidden"

I tested the webhook directly using the test messages and format from here - https://docs.microsoft.com/en-us/outlook/actionable-messages/send-via-connectors
And these work fine, so I know the webhook is operational and functioning.

Note, I have not configured Alertmanager yet, I am simply testing directly to the prometheus-teams pod to verify if it will work once alertmanager is set up.

Not sure why its telling me my connection is forbidden.

[revelnode]

I just ran a test, and if I take the formated line of the log, which makes a json like the following:

{
  "@type":"MessageCard",
  "@context":"http://schema.org/extensions",
  "themeColor":"FFA500",
  "summary":"Server High Memory usage",
  "title":"Prometheus Alert (firing)",
  "sections": [
      {
          "activityTitle":"[xxxxxxx](http://alertmanager:9093)",
          "facts": [
              {
                  "name":"description",
                  "value":"xxxxxxx"
              },
              {
                  "name":"summary",
                  "value":"Server High Memory usage"
              },
              {
                  "name":"alertname",
                  "value":"high memory load"
              },
              {
                  "name":"instance",
                  "value":"10.80.40.11:9100"
              },
              {
                  "name":"job",
                  "value":"docker nodes"
              },
              {
                  "name":"monitor",
                  "value":"master"
              },
              {
                  "name":"severity",
                  "value":"warning"
              }
            ],
          "markdown":true
      }
  ]
}

And post it directly to the webhook, it works, I see the post on channel.

[Knappek]

That is weird. Could you please try to run prometheus-msteams locally and try send a message again (this is what I am always doing)? Do this by following these steps:

1. export RUN_ARGS="-l localhost \
       -p 2000 \
       -w https://outlook.office.com/webhook/xxx/xxx \
       --log-level DEBUG"
   

2. make run-osx (for Mac, if on linux or windows, first build it with the corresponding target in the Makefile, then run it similar to that command
3. open a new tab in your terminal, navigate to ./alert/testdata and execute

   curl -X POST -d @prom_post_request.json http://localhost:2000/alertmanager
   

[Knappek]

closing this issue as I think it's due to misconfiguration. @revelnode feel free to re-open if you disagree.