Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for output_metric_tags #2160

Open
calebhailey opened this issue Oct 10, 2018 · 3 comments

Comments

@calebhailey
Copy link
Member

commented Oct 10, 2018

Expected Behavior

Add a new output_metric_tags attribute to the Check resource/specification. Allowed values should provide dot-notation access to any event attribute (e.g. entity.id, entity.organization, entity.system.os, check.name, etc). Here's an example of how it might be implemented:

type: Check
api_version: core/v2
metadata:
  namespace: default
  name: check_sensu_io
spec: 
  command: "check_http -u https://sensu.io/ -H sensu.io -N",
  runtime_assets: 
  - check_http_v0.1
  publish: true
  interval: 10
  subscriptions: 
  - docker
  output_metric_format: nagios_perfdata
  output_metric_handlers:
  - influxdb
  output_metric_tags: 
    entity: entity.name 
    namespace: entity.namespace
    app: entity.metadata.labels.application_id

Bonus points if we could support access to array objects (e.g. entity.subscriptions[0] to get the first subscription), or if we supported arbitrary tags (i.e. not matching an event attribute).

Current Behavior

No support is provided for collecting tags during metric output extraction.

Possible Solution

See description above.

Context

Currently, when using metric output extraction with the trusty Nagios check_http plugin, for a check output that looks like HTTP OK: HTTP/1.1 301 Moved Permanently - 280 bytes in 0.043 second response time |time=0.043044s;;;0.000000;10.000000 size=280B;;;0, the corresponding event will have empty tag values as shown in this event excerpt:

{
  "timestamp": 1539194388,
  "entity": {
    "class": "agent",
    "id": "46e492cdd189",
    "...": "..."
  },
  "check": {
    "name": "check_sensu_io",
    "command": "check_http -u https://sensu.io/ -H sensu.io -N",
    "output": "HTTP OK: HTTP/1.1 301 Moved Permanently - 197 bytes in 0.034 second response time |time=0.033662s;;;0.000000;10.000000 size=197B;;;0\n",
    "...": "...",
    "metrics": {
      "handlers": [
        "influxdb"
      ],
      "points": [
        {
          "name": "time",
          "value": 0.033662,
          "timestamp": 1539194388,
          "tags": []
        },
        {
          "name": "size",
          "value": 197,
          "timestamp": 1539194388,
          "tags": []
        }
      ]
    }
  }
}

The challenge with this is that when sent to InfluxDB, these appear as very generically named measurements (i.e. "time" and "size"), with no context about which system or service reported the metrics (because there are no tags). Compared with the graphite-style measurement name you would get with Sensu 1.x (e.g. 46e492cdd189.check_sensu_io.time), this leaves something to be desired, even though simple measurement names with tags would be a more native data representation in InfluxDB.

Your Environment

  • Sensu version used (sensuctl, sensu-backend, and/or sensu-agent):
  • Installation method (packages, binaries, docker etc.):
  • Operating System and version (e.g. Ubuntu 14.04):
@portertech

This comment has been minimized.

Copy link
Member

commented Oct 10, 2018

1508

@calebhailey

This comment has been minimized.

Copy link
Member Author

commented Oct 10, 2018

Commenting here to capture for posterity that @nikkiki suggested in Slack that the best way to do this is probably with Tokens... 💯

@calebhailey

This comment has been minimized.

Copy link
Member Author

commented Oct 17, 2018

In further discussion on this issue, it is worth noting that this really only impacts metrics collected in formats that don't natively support tags (e.g. Nagios PerfData and Graphite plaintext). Metrics collected in formats that support tags will have their tags extracted or "passed through".

The good/bad news about that is most of the Sensu Community Plugins have standardized around Graphite plaintext format for metrics collection.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants
You can’t perform that action at this time.