Cannot get anything to send to notifications

[mneil]

Thanks for looking at this. I've spent a good day and a half trying to make this work. No matter what settings I use, where I put the files, or how I configure the thing sends messages. I've resorted to simply trying to make it post a message to requestb.in to make sure that it will send any message. The documentation on setting up handlers is not very detailed in Sensu. The link you have to installation really just says, install a gem :). I read through the sensu handler docs and check docs. I had the default handler working to just pipe commands to cat. I have also setup other handlers and checks. But sending to Slack fails silently. I'm sure I'm missing something or have done this wrong. I would love to have more details on how to actually set this up. I am not using the Enterprise server. Here are the details.

Setup:

• Ubuntu 14.04
• Latest build of sensu server pulled from http://sensu.global.ssl.fastly.net/apt/dists/sensu/main/.
  • I'm running what was published on the 15th of July
• Ruby 2.2.0 installed separately

I tried putting the config in the handler folder. Since that didn't work I ended up with the following in /etc/sensu/config.json.

"handlers":{
    "default": {
      "type": "set",
      "handlers":[
        "slack"
      ]
    },
    "slack": {
            "type": "pipe",
            "command": "/usr/local/bin/handler-slack.rb"
        }
  },
  "slack": {
      "webhook_url": "http://requestb.in/1i40g351",
      "template" : ""
  }

again, just trying to get it to post to requestb.in.

In the sensu server logs I get messages like this

{"timestamp":"2016-06-17T18:01:50.822635+0000","level":"info","message":"handler output","handler":{"type":"pipe","command":"/usr/local/bin/handler-slack.rb","name":"slack"},"output":["only handling every 180 occurrences: test.docker/load_docker_metrics\n"]}

So I see that it is sending something, or trying to, to the slack handler. Heeding the warning, 180 occurrences, I left it on overnight. I run the check every 10 seconds. That's every 30 minutes. This morning I have nothing logged to requestb.in.

I have another check running that logs this

{"timestamp":"2016-06-17T18:03:20.708416+0000","level":"info","message":"processing event","event":{"client":{"name":"test.docker","address":"10.0.2.15","subscriptions":["docker"],"version":"0.25.1","timestamp":1466186589},"check":{"command":"check-process.rb -p cron","subscribers":["production"],"standalone":true,"interval":60,"handlers":["slack"],"name":"cron","issued":1466186603,"executed":1466186603,"duration":0.435,"output":"CheckProcess CRITICAL: Found 0 matching processes; cmd /cron/\n","status":2,"type":"standard","history":["2","2","2","2","2","2","2","2","2","2","2","2","2","2","2","2","2","2","2","2","2"],"total_state_change":0},"occurrences":1186,"action":"create","timestamp":1466186600,"id":"4e00b3aa-0a94-43c9-8108-13a173a05405","last_state_change":1466115322,"last_ok":null}}

This other check, which should send to slack, also does not trigger any messages to requestb.in. I have also left this running for over 12 hours.

Hopefully this is enough info understand what is happening and not too much info to overwhelm. I really would love to get this working. If sensu enterprise is required that is also an option for me.

[marzlarz]

Same here, spent all day trying to get this working. Worked through several of the setup issue and no longer getting error messages, but still nothing is posting to Slack. Using "proxy_address" in my slack handler config, but when running TCPDUMP I show nothing going outbound to the proxy server.... frustrated

[tpacpdan]

Took me quite some time but the config format below is working. *Edited for a formatting abomination I missed earlier

/etc/sensu/conf.d/handlers# cat slack.json

{
  "handlers": {
    "slack": {
      "type": "pipe",
     "command": "handler-slack.rb",
      "severities": [
        "ok",
        "critical",
        "unknown"
      ]
   }
  },
  "slack": {
     "webhook_url": "https://hooks.slack.com/services/#^$^#&#$^&$/",
     "proxy_address": "Proxy_Here_If_Needed",
     "proxy_port": "80"
  }
 }

[SiKing]

Same problem here: no notifications in slack, no help in logs, no help in docs. :(

I tried the configuration that @tpacpdan suggested.
I believe the handler-slack.rb is also stored in `/opt/sensu/embedded/bin/handler-slack.rb' and this path, according to Sensu docs, is automatically placed on the sensu path. Trying either path, or just the command itself with no path, gives the same results: nothing.

I also confirmed that I can ping hooks.slack.com from my server.

Out of ideas ...

[mneil]

I actually did get this to work. I will try and explain the setup for the community version I used.

Server Setup

Assuming you installed the Sensu Community server the default way on ubuntu 14.04. These docs will probably help regardless of setup, but that's what I used.

First you need to install the slack plugin (gem) sensu-install -p sensu-plugins-slack

Add the handler and config
/etc/sensu/conf.d/handlers/slack.json

{
  "handlers": {
    "slack": {
      "type": "pipe",
      "command": "handler-slack.rb",
      "filters": [
        "handle_when"
      ]
    }
  },
  "slack": {
    "webhook_url": "yourwebhookurlforslack"
  }
}

sensu knows where to find handler-slack.rb as long as you install using sensu-install

Because I found the frequency of the alerts obnoxious I created a "handle_when" filter that mimics the enterprise handle_when functionality Sensu has documented. Either add my handle when filter or remove the filters array from your slack config above. If you use the enterprise version then you can leave the config above alone and ignore my handle_when config.

/etc/sensu/conf.d/filters/handle_when.json

{
  "filters": {
    "handle_when": {
      "negate": true,
      "attributes": {
        "occurrences": "eval: ':::action:::' != 'resolve' && value != :::check.handle_when.occurrences::: && value * :::check.interval::: % :::check.handle_when.reset::: != 0"
      }
    }
  }
}

Sensu Client

In my client config I have installed the client the documented way for sensu on another Ubuntu 14.04 machine. I setup checks like:

/etc/sensu/conf.d/checks/api.json

{
  "checks": {
    "api": {
      "standalone": true,
      "handler": "slack",
      "type": "standard",
      "command": "check-process-status.sh API 8001",
      "subscribers": [
        "api"
      ],
      "refresh": 60,
      "ttl": 300,
      "interval": 60,
      "handle_when": {
        "occurrences": 3,
        "reset": 1800
      }
    }
}

My API check process is something like

/etc/sensu/plugins/check-process-status.sh

#!/bin/bash

PORT=$2
NAME=$1

PROCESS=$(supervisorctl -s "http://$( echo  $(hostname --ip-address) | sed 's/[0-9]*$/1/' ):$PORT" status)

PROCESS_UP=$( echo "$PROCESS" | grep "RUNNING" )

if [ "$PROCESS_UP" != "" ]; then
    exit 0;
fi

exit 2;

I'm running an API process with supervisord and using the check-process-status.sh file to check that the process is up. Even though supervisor is meant to keep the process running I want Sensu to notify me in the event that it is down. If the script exits with status 0 then the state is considered OK, otherwise it's 2 CRITICAL.

Sensu recommends Nagios style exit codes for state:
https://assets.nagios.com/downloads/nagioscore/docs/nagioscore/3/en/statetypes.html

To test this you could clean out my bash file and either exit 0 or 2. Tweak the handle_when filter to change how often messages are sent to slack. My setup is to send an alert after 3 concurrent states changed and then once every 30 checks afterwards until the state changes to OK.

[mneil]

Also, I went through the source and found out I could replace the slack payload with my own Ruby templates. In case anyone wants to do that you can:

Change the slack handler config on sensu server to something like:

"slack": {
    "webhook_url": "yourwebhookurlforslack",
    "message_template": "/slack-template.eruby",
    "payload_template": "/payload.eruby"
  }

You need to create those two files being referenced. Make sure those files are readable to everyone.

/slack-template.eruby

<%=
[
  @event['client']['name'] + '/' + @event['check']['name'],
  @event["check"]["output"].gsub(\'"\', \'\\"\'),
  @event["client"]["address"],
  @event["client"]["subscriptions"].join(",")
].join(" : ")
%>

/payload.eruby

{
      icon_url: slack_icon_url ? slack_icon_url : 'http://sensuapp.org/img/sensu_logo_large-c92d73db.png',
      attachments: [{
        title: "#{@event['client']['address']} - #{translate_status}",
        text: [slack_message_prefix, notice].compact.join(' '),
        color: color,
        fields: client_fields
      }]
    }
}

I believe what I put in here is what the default uses. You can customize either of these to make the message look like you want.

Note that @event is the sensu event object so you have access to anything included in your event including details from the client config. You can use that to customize the slack messages even further.

[majormoses]

@mneil glad you figured it out. I know we have documented some stuff around this a bit more but if you think there is anything we should add please open up a PR and ping me.