Skip to content

Commit

Permalink
Merge pull request #7357 from Ant1x/feature/opentsdb-tags
Browse files Browse the repository at this point in the history 
OpenTsdbWriter - Custom Tag Support
  • Loading branch information
Michael Friedrich committed Nov 15, 2019
2 parents 88c1a19 + 166e79f commit 1a503b5
Show file tree
Hide file tree
Showing 6 changed files with 420 additions and 19 deletions.
3 changes: 3 additions & 0 deletions doc/09-object-types.md
View file
Expand Up @@ -1727,6 +1727,9 @@ Configuration Attributes:
host | String | **Optional.** OpenTSDB host address. Defaults to `127.0.0.1`.
port | Number | **Optional.** OpenTSDB port. Defaults to `4242`.
enable\_ha | Boolean | **Optional.** Enable the high availability functionality. Only valid in a [cluster setup](06-distributed-monitoring.md#distributed-monitoring-high-availability-features). Defaults to `false`.
enable_generic_metrics | Boolean | **Optional.** Re-use metric names to store different perfdata values for a particular check. Use tags to distinguish perfdata instead of metric name. Defaults to `false`.
host_template | Dictionary | **Optional.** Specify additional tags to be included with host metrics. This requires a sub-dictionary named `tags`. Also specify a naming prefix by setting `metric`. More information can be found in [OpenTSDB custom tags](14-features.md#opentsdb-custom-tags) and [OpenTSDB Metric Prefix](14-features.md#opentsdb-metric-prefix). More information can be found in [OpenTSDB custom tags](14-features.md#opentsdb-custom-tags). Defaults to an `empty Dictionary`.
service_template | Dictionary | **Optional.** Specify additional tags to be included with service metrics. This requires a sub-dictionary named `tags`. Also specify a naming prefix by setting `metric`. More information can be found in [OpenTSDB custom tags](14-features.md#opentsdb-custom-tags) and [OpenTSDB Metric Prefix](14-features.md#opentsdb-metric-prefix). Defaults to an `empty Dictionary`.


### PerfdataWriter <a id="objecttype-perfdatawriter"></a>
Expand Down
184 changes: 179 additions & 5 deletions doc/14-features.md
View file
Expand Up @@ -618,20 +618,33 @@ You can enable the feature using
By default the `OpenTsdbWriter` object expects the TSD to listen at
`127.0.0.1` on port `4242`.

The current naming schema is
The current default naming schema is:

```
icinga.host.<metricname>
icinga.service.<servicename>.<metricname>
icinga.host.<perfdata_metric_label>
icinga.service.<servicename>.<perfdata_metric_label>
```

for host and service checks. The tag host is always applied.
for host and service checks. The tag `host` is always applied.

Icinga also sends perfdata warning, critical, minimum and maximum threshold values to OpenTSDB.
These are stored as new OpenTSDB metric names appended with `_warn`, `_crit`, `_min`, `_max`.
Values are only stored when the corresponding threshold exists in Icinga's perfdata.

Example:
```
icinga.service.<servicename>.<perfdata_metric_label>
icinga.service.<servicename>.<perfdata_metric_label>._warn
icinga.service.<servicename>.<perfdata_metric_label>._crit
icinga.service.<servicename>.<perfdata_metric_label>._min
icinga.service.<servicename>.<perfdata_metric_label>._max
```

To make sure Icinga 2 writes a valid metric into OpenTSDB some characters are replaced
with `_` in the target name:

```
\ (and space)
\ : (and space)
```

The resulting name in OpenTSDB might look like:
Expand Down Expand Up @@ -676,6 +689,167 @@ with the following tags
> You might want to set the tsd.core.auto_create_metrics setting to `true`
> in your opentsdb.conf configuration file.
#### OpenTSDB Metric Prefix <a id="opentsdb-metric-prefix"></a>
Functionality exists to modify the built in OpenTSDB metric names that the plugin
writes to. By default this is `icinga.host` and `icinga.service.<servicename>`.

These prefixes can be modified as necessary to any arbitary string. The prefix
configuration also supports Icinga macros, so if you rather use `<checkcommand>`
or any other variable instead of `<servicename>` you may do so.

To configure OpenTSDB metric name prefixes, create or modify the `host_template` and/or
`service_template` blocks in the `opentsdb.conf` file, to add a `metric` definition.
These modifications go hand in hand with the **OpenTSDB Custom Tag Support** detailed below,
and more information around macro use can be found there.

Additionally, using custom Metric Prefixes or your own macros in the prefix may be
helpful if you are using the **OpenTSDB Generic Metric** functionality detailed below.

An example configuration which includes prefix name modification:

```
object OpenTsdbWriter "opentsdb" {
host = "127.0.0.1"
port = 4242
host_template = {
metric = "icinga.myhost"
tags = {
location = "$host.vars.location$"
checkcommand = "$host.check_command$"
}
}
service_template = {
metric = "icinga.service.$service.check_command$"
}
}
```

The above configuration will output the following naming schema:
```
icinga.myhost.<perfdata_metric_label>
icinga.service.<check_command_name>.<perfdata_metric_label>
```
Note how `<perfdata_metric_label>` is always appended in the default naming schema mode.

#### OpenTSDB Generic Metric Naming Schema <a id="opentsdb-generic-metrics"></a>

An alternate naming schema (`Generic Metrics`) is available where OpenTSDB metric names are more generic
and do not include the Icinga perfdata label in the metric name. Instead,
perfdata labels are stored in a tag `label` which is stored along with each perfdata value.

This ultimately reduces the number of unique OpenTSDB metric names which may make
querying aggregate data easier. This also allows you to store all perfdata values for a
particular check inside one OpenTSDB metric name for each check.

This alternate naming schema can be enabled by setting the following in the OpenTSDBWriter config:
`enable_generic_metrics = true`

> **Tip**
> Consider using `Generic Metrics` along with the **OpenTSDB Metric Prefix** naming options
> described above
An example of this naming schema when compared to the default is:

```
icinga.host
icinga.service.<servicename>
```

> **Note**
> Note how `<perfdata_metric_label>` does not appear in the OpenTSDB metric name
> when using `Generic Metrics`. Instead, a new tag `label` appears on each value written
> to OpenTSDB which contains the perfdata label.
#### Custom Tags <a id="opentsdb-custom-tags"></a>

In addition to the default tags listed above, it is possible to send
your own custom tags with your data to OpenTSDB.

Note that custom tags are sent **in addition** to the default hostname,
type and service name tags. If you do not include this section in the
config file, no custom tags will be included.

Custom tags can be custom attributes or built in attributes.

Consider a host object:

```
object Host "my-server1" {
address = "10.0.0.1"
check_command = "hostalive"
vars.location = "Australia"
}
```

and a service object:

```
object Service "ping" {
host_name = "localhost"
check_command = "my-ping"
vars.ping_packets = 10
}
```

It is possible to send `vars.location` and `vars.ping_packets` along
with performance data. Additionally, any other attribute can be sent
as a tag, such as `check_command`.

You can make use of the `host_template` and `service_template` blocks
in the `opentsdb.conf` configuration file.

An example OpenTSDB configuration file which makes use of custom tags:

```
object OpenTsdbWriter "opentsdb" {
host = "127.0.0.1"
port = 4242
host_template = {
tags = {
location = "$host.vars.location$"
checkcommand = "$host.check_command$"
}
}
service_template = {
tags = {
location = "$host.vars.location$"
pingpackets = "$service.vars.ping_packets$"
checkcommand = "$service.check_command$"
}
}
}
```

Depending on what keyword the macro begins with, will determine what
attributes are available in the macro context. The below table explains
what attributes are available with links to each object type.

start of macro | description
---------------|------------------------------------------
\$host...$ | Attributes available on a [Host object](09-object-types.md#objecttype-host)
\$service...$ | Attributes available on a [Service object](09-object-types.md#objecttype-service)
\$icinga...$ | Attributes available on the [IcingaApplication object](09-object-types.md#icingaapplication)

> **Note**
>
> Ensure you do not name your custom attributes with a dot in the name.
> Dots located inside a macro tell the interpreter to expand a
> dictionary.
>
> Do not do this in your object configuration:
>
> `vars["my.attribute"]`
>
> as you will be unable to reference `my.attribute` because it is not a
> dictionary.
>
> Instead, use underscores or another character:
>
> `vars.my_attribute` or `vars["my_attribute"]`


#### OpenTSDB in Cluster HA Zones <a id="opentsdb-writer-cluster-ha"></a>

The OpenTSDB feature supports [high availability](06-distributed-monitoring.md#distributed-monitoring-high-availability-features)
Expand Down
16 changes: 16 additions & 0 deletions etc/icinga2/features-available/opentsdb.conf
View file
Expand Up @@ -6,4 +6,20 @@
object OpenTsdbWriter "opentsdb" {
//host = "127.0.0.1"
//port = 4242
//enable_generic_metrics = false

// Custom Tagging, refer to Icinga object type documentation for
// OpenTsdbWriter
//host_template = {
// metric = "icinga.host"
// tags = {
// zone = "$host.zone$"
// }
//}
//service_template = {
// metric = "icinga.service.$service.check_command$"
// tags = {
// zone = "$service.zone$"
// }
//}
}

0 comments on commit 1a503b5

Please sign in to comment.