Skip to content
main
Switch branches/tags
Code

Latest commit

 

Git stats

Files

Permalink
Failed to load latest commit information.
Type
Name
Latest commit message
Commit time
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Bonsai Asset Badge Go Test goreleaser

Sensu PagerDuty Handler

Table of Contents

Overview

The Sensu PagerDuty Handler is a Sensu Event Handler which manages PagerDuty incidents, for alerting operators. With this handler, Sensu can trigger and resolve PagerDuty incidents.

Usage examples

Help output

The Sensu Go PagerDuty handler for incident management

Usage:
  sensu-pagerduty-handler [flags]
  sensu-pagerduty-handler [command]

Available Commands:
  help        Help about any command
  version     Print the version number of this plugin

Flags:
  -k, --dedup-key-template string   The PagerDuty V2 API deduplication key template, can be set with PAGERDUTY_DEDUP_KEY_TEMPLATE (default "{{.Entity.Name}}-{{.Check.Name}}")
  -d, --details-template string     The template for the alert details, can be set with PAGERDUTY_DETAILS_TEMPLATE (default full event JSON)
  -h, --help                        help for sensu-pagerduty-handler
  -s, --status-map string           The status map used to translate a Sensu check status to a PagerDuty severity, can be set with PAGERDUTY_STATUS_MAP
  -S, --summary-template string     The template for the alert summary, can be set with PAGERDUTY_SUMMARY_TEMPLATE (default "{{.Entity.Name}}/{{.Check.Name}} : {{.Check.Output}}")
      --team string                 Envvar name for pager team(alphanumeric and underscores) holding PagerDuty V2 API authentication token, can be set with PAGERDUTY_TEAM
      --team-suffix string          Pager team suffix string to append if missing from team name, can be set with PAGERDUTY_TEAM_SUFFIX (default "_pagerduty_token")
  -t, --token string                The PagerDuty V2 API authentication token, can be set with PAGERDUTY_TOKEN

Use "sensu-pagerduty-handler [command] --help" for more information about a command.

Deduplication key

The deduplication key is determined via the --dedup-key-template argument. It is a Golang template containing the event values and defaults to {{.Entity.Name}}-{{.Check.Name}}.

PagerDuty severity mapping

Optionally you can provide mapping information between the Sensu check status and the PagerDuty incident severity. To provide the mapping you need to use the --status-map command line option or the PAGERDUTY_STATUS_MAP environment variable. The option accepts a JSON document containing the mapping information. Here's an example of the JSON document:

{
    "info": [
        0,
        1
    ],
    "warning": [
        2
    ],
    "critical:": [
        3
    ],
    "error": [
        4,
        5,
        6,
        7,
        8,
        9,
        10
    ]
}

The valid PagerDuty alert severity levels are the following:

  • info
  • warning
  • critical
  • error

Configuration

Asset registration

Sensu Assets are the best way to make use of this plugin. If you're not using an asset, please consider doing so! If you're using sensuctl 5.13 with Sensu Backend 5.13 or later, you can use the following command to add the asset:

sensuctl asset add sensu/sensu-pagerduty-handler

If you're using an earlier version of sensuctl, you can find the asset on the Bonsai Asset Index.

Handler definition

---
type: Handler
api_version: core/v2
metadata:
  name: pagerduty
  namespace: default
spec:
  type: pipe
  command: >-
    sensu-pagerduty-handler
    --dedup-key-template "{{.Entity.Namespace}}-{{.Entity.Name}}-{{.Check.Name}}"
    --status-map "{\"info\":[0],\"warning\": [1],\"critical\": [2],\"error\": [3,127]}"
    --summary-template "[{{.Entity.Namespace}}] {{.Entity.Name}}/{{.Check.Name}}: {{.Check.State}}"
    --details-template "{{.Check.Output}}\n\n{{.Check}}"
  timeout: 10
  runtime_assets:
  - sensu/sensu-pagerduty-handler
  filters:
  - is_incident
  secrets:
  - name: PAGERDUTY_TOKEN
    secret: pagerduty_authtoken

Environment variables

Most arguments for this handler are available to be set via environment variables. However, any arguments specified directly on the command line override the corresponding environment variable.

Argument Environment Variable
--token PAGERDUTY_TOKEN
--summary-template PAGERDUTY_SUMMARY_TEMPLATE
--dedup-key-template PAGERDUTY_DEDUP_KEY_TEMPLATE
--status-map PAGERDUTY_STATUS_MAP

Security Note: Care should be taken to not expose the auth token for this handler by specifying it on the command line or by directly setting the environment variable in the handler definition. It is suggested to make use of secrets management to surface it as an environment variable. The handler definition above references it as a secret. Below is an example secrets definition that make use of the built-in env secrets provider.

---
type: Secret
api_version: secrets/v1
metadata:
  name: pagerduty_token
spec:
  provider: env
  id: PAGERDUTY_TOKEN

Templates

This handler provides options for using templates to populate the values provided by the event in the message sent via SNS. More information on template syntax and format can be found in the documentation.

Argument annotations

All arguments for this handler are tunable on a per entity or check basis based on annotations. The annotations keyspace for this handler is sensu.io/plugins/sensu-pagerduty-handler/config.

NOTE: Due to check token substituion, supplying a template value such as for details-template as a check annotation requires that you place the desired template as a golang string literal (enlcosed in backticks) within another template definition. This does not apply to entity annotations.

Examples

To change the --details-template argument for a particular check, and taking into account the note above regarding templates, for that check's metadata add the following:

type: CheckConfig
api_version: core/v2
metadata:
  annotations:
    sensu.io/plugins/sensu-pagerduty-handler/config/details-template: "{{`{{.Check.Output}}`}}"
[...]

To change the --token argument for a particular check, for that checks's metadata add the following:

type: CheckConfig
api_version: core/v2
metadata:
  annotations:
    sensu.io/plugins/sensu-pagerduty-handler/config/token: abcde12345fabcd67890efabc12345de
[...]

Pager teams

Instead of specifying the authentication token directly in the check or agent annotations, you can instead reference a pager team name, which will then be used to lookup the corresponding token from the handler environment. Corresponding pager team token environment variables can be populated in the handler environment in 3 different ways

  1. Explicitly set in the handler definition
  2. Kept as Sensu secrets and referenced in the handler definition
  3. Defined in the backend service environment file, read in at backend service start.

Pager team names will be automatically suffixed with configured --team-suffix (default: _pagerduty_suffix) Note: Pager team name strings should be alphameric and underscores only. Groups of illegal characters will be mapped into a single underscore character. Ex: example-_-team will be converted to example_team

If the team token lookup fails, the explicitly provided token will be used as a fallback if available.

Example of Check Using Pager Team and Handler Environment Variables:

First set the team annotation in the check or agent resource.

Check Snippet:
---
type: CheckConfig 
api_version: core/v2 
metadata: 
  name: example-check 
  annotations: 
    sensu.io/plugins/sensu-pagerduty-handler/config/team: team_1

And define the corresponding evironment variable for the pager team's token in the handler's environment.

Handler Snippet:
---
type: Handler
api_version: core/v2
metadata:
  name: pagerduty
  namespace: default
spec:
  type: pipe
  command: >-
    sensu-pagerduty-handler
    --dedup-key-template "{{.Entity.Namespace}}-{{.Entity.Name}}-{{.Check.Name}}"
    --status-map "{\"info\":[0],\"warning\": [1],\"critical\": [2],\"error\": [3,127]}"
    --summary-template "[{{.Entity.Namespace}}] {{.Entity.Name}}/{{.Check.Name}}: {{.Check.State}}"
    --details-template "{{.Check.Output}}\n\n{{.Check}}"
  timeout: 10
  runtime_assets:
  - sensu/sensu-pagerduty-handler
  filters:
  - is_incident
  secrets:
  - name: PAGERDUTY_TOKEN
    secret: pagerduty_authtoken
  env_vars: 
  - team_1_pagerduty_token="XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
  

Proxy support

This handler supports the use of the environment variables HTTP_PROXY, HTTPS_PROXY, and NO_PROXY (or the lowercase versions thereof). HTTPS_PROXY takes precedence over HTTP_PROXY for https requests. The environment values may be either a complete URL or a "host[:port]", in which case the "http" scheme is assumed.

Installation from source

Download the latest version of the sensu-pagerduty-handler from releases, or create an executable from this source.

From the local path of the sensu-pagerduty-handler repository:

go build -o /usr/local/bin/sensu-pagerduty-handler

Contributing

See https://github.com/sensu/sensu-go/blob/master/CONTRIBUTING.md