Sensu Go Metric Tag Enrichment
Sensu Metric Tag Enrichment is a Sensu Go backend integration for enriching metric points with Entity metadata (e.g. entity name, labels), and the originating Check name by adding them as Metric Tags. Sensu Metric Tag Enrichment is useful for adding context to generic metrics collected from plugins that generate metrics in formats that don't support tags (e.g. Nagios Perfdata).
This integration provides the following features:
- Event Mutator
Sensu Go has a built-in feature called "metric extraction" that parses check output for metric data, and extract it into a generic format for processing by a Sensu Handler. The benefit of this approach is that Sensu Handlers can process metrics collected in any format without the complexity of having to directly support them – Handlers can simply consume metrics in the generic Sensu format and convert them to the desired/required format for the intended backend storage system. There are also certain performance and scalability benefits derived from performing metric extraction at the "edge" (i.e. by the Sensu Agents) as opposed to performing this conversion on the backend nodes.
Metric extraction is enabled by adding the
output_metric_format attribute to a
[Sensu Go CheckConfig][checkconfig] spec; supported formats include
However, not all metric sources are created equally. For example, Nagios plugins that emit Nagios Performance Data ("PerfData") do not support tags. Historically, it was popular to adopt a dot-notation based metric naming convention for generic metrics (e.g. a generically named
server01.http_service.response_time). Now that most modern time
series databases support custom tags, it is more practical to provide context
about collected metrics as tags (e.g.
host: server01, and
This integration provides some basic support for enriching extracted metrics with basic Entity metadata and the originating Sensu Check name.
Create a Sensu Asset named
metric-tag-enrichment(see example Asset Definition).
Create a Sensu Mutator named
metric-tag-enrichment(see example Mutator Definition).
Configure a Sensu Handler to use the Metric Tag Enrichment mutator.
spec.mutatorattribute to an existing Sensu Handler resource using
sensuctls interactive update mode:
$ sensuctl handler update influxdb ? Environment variables: ? Filters: ? Mutator: metric-tag-enrichment ? Timeout: 5 ? Type: pipe ? Runtime Assets: sensu-influxdb-handler ? Command: sensu-influxdb-handler -a http://influxdb:8086 -d 'sensu' -u 'admin' -p 'admin'
--- type: Asset api_version: core/v2 metadata: namespace: default name: metric-tag-enrichment labels: sensu.io/integration/type: backend sensu.io/integration/providers: mutator spec: url: https://github.com/calebhailey/sensu-go-metric-tag-enrichment sha512: #
--- type: Mutator api_version: core/v2 metadata: namespace: default name: metric-tag-enrichment spec: command: metric-tag-enrichment timeout: 5 runtime_assets: - metric-tag-enrichment
This project was an experiment for me to learn about Golang, so I'm approaching things very naively. If you see room for improvement or any obvious mistakes, please open an issue to help me learn! :) Thanks in advance for your feedback!
The motivation for this project was to attempt implementing a solution for an outstanding Sensu Go feature request to provide support for enriching extracted metrics with tags. Off to a good start!
- Add support for toggling the behavior of the mutator; e.g.:
--entity-nametoggle addition of an
--entity-labelstoggle addition of Entity Labels as tags (source:
--check-nametoggle addition of a
--check-labelstoggle addition of Check Labels as tags (source:
- Add support for customizing the behavior of the Entity Label and Check Label
--entity-labels region,datacenterfilter which Entity Labels are added as tags.
--check-labels application_id,teamfilter which Check Labels are added as tags.