Slack Integration

[jamiegwatkin]

Has anyone had issues with the Slack Integration?

My current configuration is as follows (webhook url obfuscated) -

notification_config {
  name: "slack"
  slack_config {
      webhook_url: "https://hooks.slack.com/services/T03Rsdfklsjdflksjdfkljskldfklsjdf8zo"
      send_resolved: true
    }
}

aggregation_rule {
  notification_config_name: "slack"
}

I've tested that the webhook url works using the following curl command -

url -X POST --data-urlencode 'payload={"text": "This is posted to <#general> and comes from *monkey-bot*.", "channel": "#general", "username": "monkey-bot", "icon_emoji": ":monkey_face:"}' https://hooks.slack.com/services/T03Rsdfklsjdflksjdfkljskldfklsjdf8zo
ok%

However I get the following error in AlertManager when a notification is triggered -

==> /var/log/alert_manager.log <==
time="2015-08-13T15:23:50Z" level=info msg="Recomputing notification outputs (active alerts have changed)" file=manager.go line=398 
time="2015-08-13T15:23:50Z" level=info msg="Sent 1 notifications" file=manager.go line=353 
time="2015-08-13T15:23:55Z" level=error msg="Error sending Slack notification: Post https://hooks.slack.com/services/T03Rsdfklsjdflksjdfkljskldfklsjdf8zo: net/http: request canceled while waiting for connection" file=notifier.go line=617 

I can see that a connection is being made but as it's HTTPS I can't see the details -

15:23:55.344214 IP 10.0.2.15.53373 > ec2-52-20-82-67.compute-1.amazonaws.com.https: Flags [S], seq 2276716265, win 29200, options [mss 1460,sackOK,TS val 316406 ecr 0,nop,wscale 6], length 0
15:23:55.383832 IP ec2-52-20-82-67.compute-1.amazonaws.com.https > 10.0.2.15.53373: Flags [S.], seq 208576001, ack 2276716266, win 65535, options [mss 1460], length 0
15:23:55.383944 IP 10.0.2.15.53373 > ec2-52-20-82-67.compute-1.amazonaws.com.https: Flags [.], ack 1, win 29200, length 0
15:23:55.385009 IP 10.0.2.15.53373 > ec2-52-20-82-67.compute-1.amazonaws.com.https: Flags [P.], seq 1:147, ack 1, win 29200, length 146
15:23:55.385207 IP ec2-52-20-82-67.compute-1.amazonaws.com.https > 10.0.2.15.53373: Flags [.], ack 147, win 65535, length 0
15:23:55.426374 IP ec2-52-20-82-67.compute-1.amazonaws.com.https > 10.0.2.15.53373: Flags [P.], seq 1:1229, ack 147, win 65535, length 1228
15:23:55.426406 IP 10.0.2.15.53373 > ec2-52-20-82-67.compute-1.amazonaws.com.https: Flags [.], ack 1229, win 31928, length 0
15:23:55.426442 IP ec2-52-20-82-67.compute-1.amazonaws.com.https > 10.0.2.15.53373: Flags [P.], seq 1229:2457, ack 147, win 65535, length 1228
15:23:55.426450 IP 10.0.2.15.53373 > ec2-52-20-82-67.compute-1.amazonaws.com.https: Flags [.], ack 2457, win 34384, length 0
15:23:55.426469 IP ec2-52-20-82-67.compute-1.amazonaws.com.https > 10.0.2.15.53373: Flags [P.], seq 2457:2833, ack 147, win 65535, length 376
15:23:55.426472 IP 10.0.2.15.53373 > ec2-52-20-82-67.compute-1.amazonaws.com.https: Flags [.], ack 2833, win 36840, length 0
15:23:55.446903 IP 10.0.2.15.53373 > ec2-52-20-82-67.compute-1.amazonaws.com.https: Flags [P.], seq 147:222, ack 2833, win 36840, length 75
15:23:55.447114 IP ec2-52-20-82-67.compute-1.amazonaws.com.https > 10.0.2.15.53373: Flags [.], ack 222, win 65535, length 0
15:23:55.447266 IP 10.0.2.15.53373 > ec2-52-20-82-67.compute-1.amazonaws.com.https: Flags [P.], seq 222:228, ack 2833, win 36840, length 6
15:23:55.447355 IP 10.0.2.15.53373 > ec2-52-20-82-67.compute-1.amazonaws.com.https: Flags [P.], seq 228:273, ack 2833, win 36840, length 45
15:23:55.447420 IP ec2-52-20-82-67.compute-1.amazonaws.com.https > 10.0.2.15.53373: Flags [.], ack 228, win 65535, length 0
15:23:55.447472 IP ec2-52-20-82-67.compute-1.amazonaws.com.https > 10.0.2.15.53373: Flags [.], ack 273, win 65535, length 0
15:23:55.567803 IP ec2-52-20-82-67.compute-1.amazonaws.com.https > 10.0.2.15.53373: Flags [P.], seq 2833:2884, ack 273, win 65535, length 51
15:23:55.606955 IP 10.0.2.15.53373 > ec2-52-20-82-67.compute-1.amazonaws.com.https: Flags [.], ack 2884, win 36840, length 0
15:24:25.567987 IP 10.0.2.15.53373 > ec2-52-20-82-67.compute-1.amazonaws.com.https: Flags [.], ack 2884, win 36840, length 0
15:24:25.568260 IP ec2-52-20-82-67.compute-1.amazonaws.com.https > 10.0.2.15.53373: Flags [.], ack 273, win 65535, length 0
15:24:55.670983 IP 10.0.2.15.53373 > ec2-52-20-82-67.compute-1.amazonaws.com.https: Flags [.], ack 2884, win 36840, length 0
15:24:55.671307 IP ec2-52-20-82-67.compute-1.amazonaws.com.https > 10.0.2.15.53373: Flags [.], ack 273, win 65535, length 0
15:24:55.754572 IP ec2-52-20-82-67.compute-1.amazonaws.com.https > 10.0.2.15.53373: Flags [P.], seq 2884:2915, ack 273, win 65535, length 31
15:24:55.754614 IP 10.0.2.15.53373 > ec2-52-20-82-67.compute-1.amazonaws.com.https: Flags [.], ack 2915, win 36840, length 0
15:24:55.754840 IP 10.0.2.15.53373 > ec2-52-20-82-67.compute-1.amazonaws.com.https: Flags [P.], seq 273:304, ack 2915, win 36840, length 31
15:24:55.754916 IP 10.0.2.15.53373 > ec2-52-20-82-67.compute-1.amazonaws.com.https: Flags [F.], seq 304, ack 2915, win 36840, length 0
15:24:55.755011 IP ec2-52-20-82-67.compute-1.amazonaws.com.https > 10.0.2.15.53373: Flags [.], ack 304, win 65535, length 0
15:24:55.755017 IP ec2-52-20-82-67.compute-1.amazonaws.com.https > 10.0.2.15.53373: Flags [.], ack 305, win 65535, length 0
15:24:55.794813 IP ec2-52-20-82-67.compute-1.amazonaws.com.https > 10.0.2.15.53373: Flags [F.], seq 2915, ack 305, win 65535, length 0
15:24:55.794863 IP 10.0.2.15.53373 > ec2-52-20-82-67.compute-1.amazonaws.com.https: Flags [.], ack 2916, win 36840, length 0

I've tried setting 'channel' in slack_config to both my '@mynickname' and '#mychannel' but this didn't seem to fix the issue.

I've looked around and I can't see many examples of the Slack integration which makes me think that it might actually be broken.

Any help is much appreciated!

[juliusv]

/cc-ing all the people who I know touched/used the Slack integration: @farcaller @miekg @sdurrheimer. Maybe someone of you has an idea?

[miekg]

This is working for us in production, similar config, alerts are delivered.

Naive question: have you done the curl from the same server?

[jamiegwatkin]

@miekg I did curl on the same server and got a 200 OK along with the message appearing in the Slack channel.

I should have probably mentioned this earlier but I was using the release - alertmanager-0.0.3.linux-amd64.tar.gz.

After grabbing the latest source and building it I was able to send a notification with Slack.

I'll stick to bleeding edge from now on!