New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Event filtering in sensu-plugin is deprecated causing the handler not to always work #52

Open
pietervogelaar opened this Issue Sep 27, 2017 · 14 comments

Comments

Projects
None yet
4 participants
@pietervogelaar

pietervogelaar commented Sep 27, 2017

I experience that the slack handler does not always seem to work. In the sensu-server.log I see:

{
  "timestamp": "2017-09-27T11:47:34.157563+0200",
  "level": "info",
  "message": "handler output",
  "handler": {
    "command": "handler-slack.rb",
    "type": "pipe",
    "filters": [
      "production_environment",
      "state_change_with_occurrences_critical"
    ],
    "severities": [
      "ok",
      "critical"
    ],
    "handle_flapping": false,
    "handle_silenced": false,
    "name": "slack"
  },
  "event": {
    "id": "e47824be-0e42-4586-8944-77c946ab84c3"
  },
  "output": [
    "warning: event filtering in sensu-plugin is deprecated, see http://bit.ly/sensu-plugin\nconnection refused attempting to query the sensu api for a stash\nconnection refused attempting to query the sensu api for a stash\nconnection refused attempting to query the sensu api for a stash\nwarning: occurrence filtering in sensu-plugin is deprecated, see http://bit.ly/sensu-plugin\nonly handling every 60 occurrences: myexamplehost.org/check_http_example_api\n"
  ]
}

In cases that the handler DOES work I see also the client and check hash in the hash above. How can this be fixed?

@majormoses

This comment has been minimized.

Show comment
Hide comment
@majormoses

majormoses Oct 7, 2017

Member

We need to remove the filtering from the slack handler per: https://blog.sensuapp.org/deprecating-event-filtering-in-sensu-plugin-b60c7c500be3

Member

majormoses commented Oct 7, 2017

We need to remove the filtering from the slack handler per: https://blog.sensuapp.org/deprecating-event-filtering-in-sensu-plugin-b60c7c500be3

@vegardx

This comment has been minimized.

Show comment
Hide comment
@vegardx

vegardx Jan 10, 2018

Any update on this? Seems related to the issues I'm seeing when using payload_template. As soon as I remove the payload_template the messages are sent to Slack. Nothing in the logs, whatsoever, except for the part about deprecated event filtering.

vegardx commented Jan 10, 2018

Any update on this? Seems related to the issues I'm seeing when using payload_template. As soon as I remove the payload_template the messages are sent to Slack. Nothing in the logs, whatsoever, except for the part about deprecated event filtering.

@pietervogelaar

This comment has been minimized.

Show comment
Hide comment
@pietervogelaar

pietervogelaar Jan 10, 2018

As a workaround I created my own Slack handler in python which works great:

#!/usr/bin/env python

import argparse
import httplib
try:
    import simplejson as json
except ImportError:
    import json
import os
import sys
import traceback

try:
    import requests
    requests.packages.urllib3.disable_warnings()
except ImportError:
    raise ImportError('Missing dependency "requests". \
        Do ``pip install requests``.')

try:
    import yaml
except ImportError:
    raise ImportError('Missing dependency "pyyaml". \
        Do ``pip install pyyaml``.')

OK_CODES = [httplib.OK, httplib.CREATED, httplib.ACCEPTED, httplib.CONFLICT]
UNREACHABLE_CODES = [httplib.NOT_FOUND]
WEBHOOK_TOKEN = None
DASHBOARD_URL = None


def _post_webhook(body, verbose=False):
    url = 'https://hooks.slack.com/services/{}'.format(WEBHOOK_TOKEN)

    headers = {
        'Content-Type': 'application/json; charset=utf-8',
    }

    try:
        if verbose:
            print('Webhook POST: url: %s, headers: %s, body: %s\n' % (url, headers, body))
        r = requests.post(url, data=json.dumps(body), headers=headers, verify=True)
    except:
        raise Exception('Cannot connect to Slack endpoint %s.' % url)
    else:
        status = r.status_code

        if status in UNREACHABLE_CODES:
            msg = 'Webhook URL %s does not exist. Check Slack documentation!' % (url)
            raise Exception(msg)

        if status not in OK_CODES:
            sys.stderr.write('Failed posting Sensu event to Slack. HTTP_CODE: \
                %d\n' % status)
        else:
            sys.stdout.write('Sent Sensu event to Slack. HTTP_CODE: \
                %d\n' % status)


def _post_event_to_slack(payload, verbose=False):

    if payload['check']['status'] == 0:
        status = 'OK'
        color = '#36A64F'
    elif payload['check']['status'] == 1:
        status = 'WARNING'
        color = '#FFCC00'
    elif payload['check']['status'] == 2:
        status = 'CRITICAL'
        color = '#FF0000'
    else:
        status = 'UNKNOWN'
        color = '#6600CC'

    title = '<{0}/{1}?check={2}|{1}/{2}> - {3}'.format(DASHBOARD_URL,
                                                       payload['client']['name'],
                                                       payload['check']['name'],
                                                       status)

    text = payload['check']['output']
    client = '{0} ({1})'.format(payload['client']['name'], payload['client']['address'])

    if 'environment' in payload['client']:
        environment = payload['client']['environment']
    else:
        environment = None

    if 'class' in payload['client']:
        classification = payload['client']['class']
    else:
        classification = None

    body = {
        'icon_url': 'https://avatars2.githubusercontent.com/u/10713628?v=4&s=400',
        'attachments': [
            {
                'title': title,
                'text': text,
                'color': color,
                'fields': [
                    {
                        'title': 'client',
                        'value': client,
                        'short': False
                    },
                    {
                        'title': 'environment',
                        'value': environment,
                        'short': True
                    },
                    {
                        'title': 'class',
                        'value': classification,
                        'short': True
                    }
                ]
            }
        ],
        'username': 'sensu'
    }

    try:
        _post_webhook(body=body, verbose=verbose)
        return True
    except:
        traceback.print_exc(limit=20)
        print('Cannot send event to Slack')
        sys.exit(4)

    return False


def _set_config_opts(config_file, verbose=False):
    global WEBHOOK_TOKEN
    global DASHBOARD_URL

    if not os.path.exists(config_file):
        print('Configuration file %s not found. Exiting!!!' % config_file)
        sys.exit(1)

    with open(config_file) as f:
        config = yaml.safe_load(f)

        if verbose:
            print('Contents of config file: %s' % config)

        WEBHOOK_TOKEN = config['webhook_token']
        DASHBOARD_URL = config['dashboard_url']


def main(config_file, payload, verbose=False):
    _set_config_opts(config_file=config_file, verbose=verbose)
    _post_event_to_slack(payload, verbose=verbose)


if __name__ == '__main__':
    parser = argparse.ArgumentParser(description='StackStorm sensu event handler.')
    parser.add_argument('config_path',
                        help='Exchange to listen on')
    parser.add_argument('--verbose', '-v', required=False, action='store_true',
                        help='Verbose mode.')
    args = parser.parse_args()
    payload = sys.stdin.read().strip()

    try:
        payload = json.loads(payload)
    except:
        print('Invalid JSON payload %s.' % payload)
        sys.exit(3)

    try:
        client = payload['client']['name']
        check = payload['check']['name']
    except KeyError:
        print('Invalid payload spec %s.' % payload)

    main(config_file=args.config_path, payload=payload, verbose=args.verbose)

pietervogelaar commented Jan 10, 2018

As a workaround I created my own Slack handler in python which works great:

#!/usr/bin/env python

import argparse
import httplib
try:
    import simplejson as json
except ImportError:
    import json
import os
import sys
import traceback

try:
    import requests
    requests.packages.urllib3.disable_warnings()
except ImportError:
    raise ImportError('Missing dependency "requests". \
        Do ``pip install requests``.')

try:
    import yaml
except ImportError:
    raise ImportError('Missing dependency "pyyaml". \
        Do ``pip install pyyaml``.')

OK_CODES = [httplib.OK, httplib.CREATED, httplib.ACCEPTED, httplib.CONFLICT]
UNREACHABLE_CODES = [httplib.NOT_FOUND]
WEBHOOK_TOKEN = None
DASHBOARD_URL = None


def _post_webhook(body, verbose=False):
    url = 'https://hooks.slack.com/services/{}'.format(WEBHOOK_TOKEN)

    headers = {
        'Content-Type': 'application/json; charset=utf-8',
    }

    try:
        if verbose:
            print('Webhook POST: url: %s, headers: %s, body: %s\n' % (url, headers, body))
        r = requests.post(url, data=json.dumps(body), headers=headers, verify=True)
    except:
        raise Exception('Cannot connect to Slack endpoint %s.' % url)
    else:
        status = r.status_code

        if status in UNREACHABLE_CODES:
            msg = 'Webhook URL %s does not exist. Check Slack documentation!' % (url)
            raise Exception(msg)

        if status not in OK_CODES:
            sys.stderr.write('Failed posting Sensu event to Slack. HTTP_CODE: \
                %d\n' % status)
        else:
            sys.stdout.write('Sent Sensu event to Slack. HTTP_CODE: \
                %d\n' % status)


def _post_event_to_slack(payload, verbose=False):

    if payload['check']['status'] == 0:
        status = 'OK'
        color = '#36A64F'
    elif payload['check']['status'] == 1:
        status = 'WARNING'
        color = '#FFCC00'
    elif payload['check']['status'] == 2:
        status = 'CRITICAL'
        color = '#FF0000'
    else:
        status = 'UNKNOWN'
        color = '#6600CC'

    title = '<{0}/{1}?check={2}|{1}/{2}> - {3}'.format(DASHBOARD_URL,
                                                       payload['client']['name'],
                                                       payload['check']['name'],
                                                       status)

    text = payload['check']['output']
    client = '{0} ({1})'.format(payload['client']['name'], payload['client']['address'])

    if 'environment' in payload['client']:
        environment = payload['client']['environment']
    else:
        environment = None

    if 'class' in payload['client']:
        classification = payload['client']['class']
    else:
        classification = None

    body = {
        'icon_url': 'https://avatars2.githubusercontent.com/u/10713628?v=4&s=400',
        'attachments': [
            {
                'title': title,
                'text': text,
                'color': color,
                'fields': [
                    {
                        'title': 'client',
                        'value': client,
                        'short': False
                    },
                    {
                        'title': 'environment',
                        'value': environment,
                        'short': True
                    },
                    {
                        'title': 'class',
                        'value': classification,
                        'short': True
                    }
                ]
            }
        ],
        'username': 'sensu'
    }

    try:
        _post_webhook(body=body, verbose=verbose)
        return True
    except:
        traceback.print_exc(limit=20)
        print('Cannot send event to Slack')
        sys.exit(4)

    return False


def _set_config_opts(config_file, verbose=False):
    global WEBHOOK_TOKEN
    global DASHBOARD_URL

    if not os.path.exists(config_file):
        print('Configuration file %s not found. Exiting!!!' % config_file)
        sys.exit(1)

    with open(config_file) as f:
        config = yaml.safe_load(f)

        if verbose:
            print('Contents of config file: %s' % config)

        WEBHOOK_TOKEN = config['webhook_token']
        DASHBOARD_URL = config['dashboard_url']


def main(config_file, payload, verbose=False):
    _set_config_opts(config_file=config_file, verbose=verbose)
    _post_event_to_slack(payload, verbose=verbose)


if __name__ == '__main__':
    parser = argparse.ArgumentParser(description='StackStorm sensu event handler.')
    parser.add_argument('config_path',
                        help='Exchange to listen on')
    parser.add_argument('--verbose', '-v', required=False, action='store_true',
                        help='Verbose mode.')
    args = parser.parse_args()
    payload = sys.stdin.read().strip()

    try:
        payload = json.loads(payload)
    except:
        print('Invalid JSON payload %s.' % payload)
        sys.exit(3)

    try:
        client = payload['client']['name']
        check = payload['check']['name']
    except KeyError:
        print('Invalid payload spec %s.' % payload)

    main(config_file=args.config_path, payload=payload, verbose=args.verbose)
@majormoses

This comment has been minimized.

Show comment
Hide comment
@majormoses

majormoses Jan 11, 2018

Member

That is indeed strange, I have not had the time to look more into this as I don't use slack at our work. I can't think of why the filtering would change if you specify a template. I will try to take a quick peak and see if see if anything jumps out.

Member

majormoses commented Jan 11, 2018

That is indeed strange, I have not had the time to look more into this as I don't use slack at our work. I can't think of why the filtering would change if you specify a template. I will try to take a quick peak and see if see if anything jumps out.

@majormoses

This comment has been minimized.

Show comment
Hide comment
@majormoses

majormoses Jan 11, 2018

Member

so re-reading through the original message I think this might be able to be resolved by bumping the dependency of sensu-plugin to 2.x and it might solve our problems. I will put together a PR an link it here. If someone can test it out and it works we can merge and release.

Member

majormoses commented Jan 11, 2018

so re-reading through the original message I think this might be able to be resolved by bumping the dependency of sensu-plugin to 2.x and it might solve our problems. I will put together a PR an link it here. If someone can test it out and it works we can merge and release.

@majormoses majormoses referenced this issue Jan 11, 2018

Merged

Bump dependency of `sensu-plugin` to 2.x #58

0 of 5 tasks complete
@vegardx

This comment has been minimized.

Show comment
Hide comment
@vegardx

vegardx Jan 11, 2018

Thanks, I'll build it and try it today!

vegardx commented Jan 11, 2018

Thanks, I'll build it and try it today!

@vegardx

This comment has been minimized.

Show comment
Hide comment
@vegardx

vegardx Jan 11, 2018

Unless I'm doing something wrong it seems like events are still being filtered. Is there anything in particular you'd like to see as for logs? I'm doing this on a quite busy server, so logging can be a little hard to dissect or follow, so I might be overlooking something.

vegardx commented Jan 11, 2018

Unless I'm doing something wrong it seems like events are still being filtered. Is there anything in particular you'd like to see as for logs? I'm doing this on a quite busy server, so logging can be a little hard to dissect or follow, so I might be overlooking something.

@majormoses

This comment has been minimized.

Show comment
Hide comment
@majormoses

majormoses Jan 11, 2018

Member

Hmm, the only way I can explain that would be if you have defined in your handler some filter such as occurrences.

Member

majormoses commented Jan 11, 2018

Hmm, the only way I can explain that would be if you have defined in your handler some filter such as occurrences.

@majormoses

This comment has been minimized.

Show comment
Hide comment
@majormoses

majormoses Jan 11, 2018

Member

Is it the same message as before?

Member

majormoses commented Jan 11, 2018

Is it the same message as before?

@majormoses majormoses closed this in #58 Jan 12, 2018

@majormoses majormoses reopened this Jan 12, 2018

@majormoses

This comment has been minimized.

Show comment
Hide comment
@majormoses

majormoses Jan 12, 2018

Member

Re-opening as @vegardx says the change did not address their problem, if you could try with the released gem and let us know. In addition if I could see the handler config for slack and if there are any filters applied please include them.

Member

majormoses commented Jan 12, 2018

Re-opening as @vegardx says the change did not address their problem, if you could try with the released gem and let us know. In addition if I could see the handler config for slack and if there are any filters applied please include them.

@windowsrefund

This comment has been minimized.

Show comment
Hide comment
@windowsrefund

windowsrefund Jan 16, 2018

Using 3.0.0 now but handler-slack-multichannel.rb is not working.

{"timestamp":"2018-01-16T14:48:56.939399-0500","level":"error","message":"handler output","handler":{"command":"handler-slack-multichannel.rb","type":"pipe","name":"slack"},"event":{"id":"4ed44eb7-e8dd-4176-80bb-72e0afbf68bb"},"output":["/opt/sensu/embedded/lib/ruby/gems/2.4.0/gems/sensu-plugins-slack-3.0.0/bin/handler-slack-multichannel.rb:171:incompile_channel_list': undefined method join' for \"ops-alerts\":String (NoMethodError)\n\tfrom /opt/sensu/embedded/lib/ruby/gems/2.4.0/gems/sensu-plugins-slack-3.0.0/bin/handler-slack-multichannel.rb:185:inslack_channels'\n\tfrom /opt/sensu/embedded/lib/ruby/gems/2.4.0/gems/sensu-plugins-slack-3.0.0/bin/handler-slack-multichannel.rb:205:in handle'\n\tfrom /opt/sensu/embedded/lib/ruby/gems/2.4.0/gems/sensu-plugin-2.3.0/lib/sensu-handler.rb:77:inblock in class:Handler'\n"]}`

windowsrefund commented Jan 16, 2018

Using 3.0.0 now but handler-slack-multichannel.rb is not working.

{"timestamp":"2018-01-16T14:48:56.939399-0500","level":"error","message":"handler output","handler":{"command":"handler-slack-multichannel.rb","type":"pipe","name":"slack"},"event":{"id":"4ed44eb7-e8dd-4176-80bb-72e0afbf68bb"},"output":["/opt/sensu/embedded/lib/ruby/gems/2.4.0/gems/sensu-plugins-slack-3.0.0/bin/handler-slack-multichannel.rb:171:incompile_channel_list': undefined method join' for \"ops-alerts\":String (NoMethodError)\n\tfrom /opt/sensu/embedded/lib/ruby/gems/2.4.0/gems/sensu-plugins-slack-3.0.0/bin/handler-slack-multichannel.rb:185:inslack_channels'\n\tfrom /opt/sensu/embedded/lib/ruby/gems/2.4.0/gems/sensu-plugins-slack-3.0.0/bin/handler-slack-multichannel.rb:205:in handle'\n\tfrom /opt/sensu/embedded/lib/ruby/gems/2.4.0/gems/sensu-plugin-2.3.0/lib/sensu-handler.rb:77:inblock in class:Handler'\n"]}`

@majormoses

This comment has been minimized.

Show comment
Hide comment
@majormoses

majormoses Jan 16, 2018

Member

Did you update a working setup or is this a new one? The reason I ask is that this error indicates a configuration error: https://github.com/sensu-plugins/sensu-plugins-slack/blob/3.0.0/bin/handler-slack-multichannel.rb#L169-L172. Specifically the line it is breaking on is here: https://github.com/sensu-plugins/sensu-plugins-slack/blob/3.0.0/bin/handler-slack-multichannel.rb#L171 which only triggers if you do not have any client or check configured channels and is a string rather than an array.

$ irb
irb(main):001:0> a = 'ops-alerts'
=> "ops-alerts"
irb(main):002:0> a.join(',')
NoMethodError: undefined method `join' for "ops-alerts":String
	from (irb):2
	from /var/lib/jenkins/.rbenv/versions/2.3.5/bin/irb:11:in `<main>'
irb(main):003:0> b = ['ops-alerts']
=> ["ops-alerts"]
irb(main):004:0> b.join(',')
=> "ops-alerts"

We could probably put in some extra validation in this function to help more easily surface the issue: https://github.com/sensu-plugins/sensu-plugins-slack/blob/3.0.0/bin/handler-slack-multichannel.rb#L160

Member

majormoses commented Jan 16, 2018

Did you update a working setup or is this a new one? The reason I ask is that this error indicates a configuration error: https://github.com/sensu-plugins/sensu-plugins-slack/blob/3.0.0/bin/handler-slack-multichannel.rb#L169-L172. Specifically the line it is breaking on is here: https://github.com/sensu-plugins/sensu-plugins-slack/blob/3.0.0/bin/handler-slack-multichannel.rb#L171 which only triggers if you do not have any client or check configured channels and is a string rather than an array.

$ irb
irb(main):001:0> a = 'ops-alerts'
=> "ops-alerts"
irb(main):002:0> a.join(',')
NoMethodError: undefined method `join' for "ops-alerts":String
	from (irb):2
	from /var/lib/jenkins/.rbenv/versions/2.3.5/bin/irb:11:in `<main>'
irb(main):003:0> b = ['ops-alerts']
=> ["ops-alerts"]
irb(main):004:0> b.join(',')
=> "ops-alerts"

We could probably put in some extra validation in this function to help more easily surface the issue: https://github.com/sensu-plugins/sensu-plugins-slack/blob/3.0.0/bin/handler-slack-multichannel.rb#L160

@windowsrefund

This comment has been minimized.

Show comment
Hide comment
@windowsrefund

windowsrefund Jan 17, 2018

The setup worked with 2.0.0. Here is my /etc/sensu/conf.d/slack.json

{
  "slack": {
    "channels": {
      "default": "ops-alerts"
    },
    "webhook_urls": {
      "foo": "https://....",
      "bar": "https://...",
      "baz": "https://...",
      "ops-alerts": "https://..."
    }
  }
}

As you suggested, changing the default to a list rather than a string did fix the issue. Thank you.

windowsrefund commented Jan 17, 2018

The setup worked with 2.0.0. Here is my /etc/sensu/conf.d/slack.json

{
  "slack": {
    "channels": {
      "default": "ops-alerts"
    },
    "webhook_urls": {
      "foo": "https://....",
      "bar": "https://...",
      "baz": "https://...",
      "ops-alerts": "https://..."
    }
  }
}

As you suggested, changing the default to a list rather than a string did fix the issue. Thank you.

@majormoses

This comment has been minimized.

Show comment
Hide comment
@majormoses

majormoses Jan 17, 2018

Member

OK it looks like someone assumed that it should always be an array at some point we can probably make it so it supports either. I will not have time to work on this until maybe the weekend but maybe I can get something for you to test by next week. In the meantime I guess you should downgrade to 2.0 I am still not sure why you did not hit this with 2.0 but maybe something in the upstream sensu-plugin library caused those to be swallowed.

Member

majormoses commented Jan 17, 2018

OK it looks like someone assumed that it should always be an array at some point we can probably make it so it supports either. I will not have time to work on this until maybe the weekend but maybe I can get something for you to test by next week. In the meantime I guess you should downgrade to 2.0 I am still not sure why you did not hit this with 2.0 but maybe something in the upstream sensu-plugin library caused those to be swallowed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment