Skip to content
Sensu Go handler for implementing "self healing" workflows
Go Shell
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
testing BREAKING CHANGE: use io.sensu.remediation.config.actions annotation (#6) Oct 2, 2019
.bonsai.yml Add CI via Travis (#7) Oct 8, 2019
.gitignore
.goreleaser.yml remove freebsd, netbsd and solaris from goreleaser Oct 25, 2019
.travis.yml
LICENSE
README.md
build.sh
go.mod
go.sum
main.go Revised readme, rename project and fix CI (#9) Oct 25, 2019

README.md

Sensu Remediation Handler

Bonsai Asset Badge Build Status

Overview

The Sensu Remediation Handler is a Sensu Go event handler that enables you to build self-healing workflows in Sensu.

The Sensu Remediation Handler – and other similar self-healing workflows in Sensu – combine a few Sensu features:

  • An "unscheduled check" configuration (i.e. a Sensu check with the "publish": false attribute set).
  • The Sensu agent's built-in entity subscriptions (e.g. entity:web-server-01), which make it possible to target a single agent with a check execution request.
  • The Sensu Checks API POST /checks/:check/execute endpoint, which allows this handler to issue ad hoc check requests.

Configuration

Environment Variables

The Sensu Remediation Handler does not honor any command line flags. Instead, it requires environment variables, either in the handler definition or in the Sensu backend service environment.

SENSU_API_URL
description URL for Sensu backend, including scheme, hostname or IP address, and port.
required false
type String
default http://127.0.0.1:8080
example SENSU_API_URL=http://sensu.example.com:8080
SENSU_USER
description Username for the Sensu API.
required true
type String
example SENSU_USER=remediation-handler
SENSU_PASS
description Password for the Sensu API.
required true
type String
example SENSU_PASS=setecastronomy
SENSU_API_CERT_FILE
description Filesystem path to certificate authority (CA) certificate used to validate https Sensu API connections.
required false
type String
example SENSU_API_CERT_FILE=/etc/sensu/cacert.pem

Annotations

Although environment variables provide connection details for the Sensu API, you'll use the io.sensu.remediation.config.actions check annotation to provide the configuration that defines remediation activities.

Annotation Specification

The Sensu Remediation Handler uses the string value of the io.sensu.remediation.config.actions check annotation to determine which remediation actions, if any, should be scheduled for a given event.

When present, the value of the io.sensu.remediation.config.actions check annotation must be a array of objects containing key/value pairs. Each object element in the array must conform to the remediation action specification.

Remediation Action Specification

description
description A human-readable representation of the remediation action.
required false
type String
example "description": "restart failed ntpd service"
request
description The name of the check to be scheduled by the remediation action.
required true
type String
example "request": "remediate-ntpd-service"
occurrences
description A list of occurrence counts at which the remediation action is triggered.
required true
type Array of integers
example "occurrences": [4,14,42]
severities
description A list of check status severities that are allowed for the remediation action.
required true
type Array of integers
example "severities": [1]
subscriptions
description A list of agent subscriptions for targeting remediation actions.
required true
type Array of strings
example "subscriptions": ["ntpd"]

Setup

  1. Create a dedicated Sensu user and role for the remediation handler.

    sensuctl role create remediation-handler --namespace=default --verb=create,update --resource checks
    sensuctl role-binding create remediation-handler --role=remediation-handler --user=remediation-handler
    sensuctl user create remediation-handler --password REPLACEME
  2. Register the remediation handler asset.

    sensuctl asset add sensu/sensu-remediation-handler --rename sensu-remediation-handler
  3. Configure the remediation handler.

    ---
    type: Handler
    api_version: core/v2
    metadata:
      name: remediation
      namespace: default
    spec:
      type: pipe
      command: sensu-remediation-handler
      timeout: 10
      runtime_assets:
      - sensu-remediation-handler
      env_vars:
      - "SENSU_API_URL=http://127.0.0.1:8080"
      - "SENSU_API_CERT_FILE="
      - "SENSU_API_USER=remediation-handler"
      - "SENSU_API_PASS=REPLACEME"

    Save this definition to a file named sensu-remediation-handler.yaml and run:

    sensuctl create -f sensu-remediation-handler.yaml

Examples

Example "Unscheduled" Check (Remediation Action)

---
type: CheckConfig
api_version: core/v2
metadata:
  name: systemd-start-nginx
  namespace: default
spec:
  command: sudo systemctl start nginx
  publish: false
  interval: 10 # interval is required but not used
  subscriptions: []

Example Check Definition and Remediation Request Configuration

---
type: CheckConfig
api_version: core/v2
metadata:
  name: check-nginx
  namespace: default
  labels:
    foo: bar
  annotations:
    io.sensu.remediation.config.actions: |
      [
        {
          "description": "Perform this action once after Nginx has been down for 30 seconds.",
          "request": "systemd-start-nginx",
          "occurrences": [ 3 ],
          "severities": [ 1,2 ]
        },
        {
          "description": "Perform this action once after Nginx has been down for 120 seconds.",
          "request": "systemd-restart-nginx",
          "occurrences": [ 12 ],
          "severities": [ 1,2 ]
        }
      ]
spec:
  command: check_http -H 127.0.0.1 -P 80 -N
  publish: true
  interval: 10
  handlers:
  - remediation
  subscriptions:
  - nginx

Acknowledgements

This handler implements a pattern first implemented in Nick Stielau's Sensu Remediator circa 2012. Thanks, Nick!

You can’t perform that action at this time.