Skip to content

Commit

Permalink
fix: allow dropping metrics, labels and configuring histogram bucket …
Browse files Browse the repository at this point in the history
…boundaries to avoid high cardinality (#974)

Signed-off-by: Rodrigo Fior Kuntzer <rodrigo@miro.com>
Co-authored-by: shuting <shuting@nirmata.com>
  • Loading branch information
rodrigorfk and realshuting committed Oct 13, 2023
1 parent da9d0b0 commit 4ee00e5
Showing 1 changed file with 25 additions and 11 deletions.
36 changes: 25 additions & 11 deletions content/en/docs/Monitoring/_index.md
Expand Up @@ -119,28 +119,42 @@ reportsController:

## Configuring the metrics

While installing Kyverno via Helm, you also have the ability to configure which metrics to expose.
While installing Kyverno via Helm, you also have the ability to configure which metrics to expose, this configuration will be stored in the `kyverno-metrics` ConfigMap.

You can configure which Namespaces you want to `include` and/or `exclude` for metric exportation when configuring your Helm chart. This configuration is useful in situations where you might want to exclude the exposure of Kyverno metrics for certain Namespaces like test or experimental Namespaces. Likewise, you can include certain Namespaces if you want to monitor Kyverno-related activity for only a set of certain critical Namespaces. Exporting the right set of Namespaces (as opposed to exposing all Namespaces) can end up substantially reducing the memory footprint of Kyverno's metrics exporter.
You can configure which Namespaces you want to `include` and/or `exclude` for metric exportation when configuring your Helm chart. This configuration is useful in situations where you might want to exclude the exposure of Kyverno metrics for certain Namespaces like test or experimental Namespaces. Likewise, you can include certain Namespaces if you want to monitor Kyverno-related activity for only a set of certain critical Namespaces. Exporting the right set of Namespaces (as opposed to exposing all Namespaces) can end up substantially reducing the memory footprint of Kyverno's metrics exporter. Moreover, you can also configure the exposure of specific metrics, disabling them completely or dropping some label dimensions. For Histograms, you can change the default bucket boundaries or configure it for a specific metric as well.

```sh
```yaml
...
metricsConfig:
namespaces: {
"include": [],
"exclude": []
}
# 'namespaces.include': list of namespaces to capture metrics for. Default: all namespaces included.
# 'namespaces.exclude': list of namespaces to NOT capture metrics for. Default: [], none of the namespaces excluded.
# 'namespaces.include': list of namespaces to capture metrics for. Default: all namespaces included.
# 'namespaces.exclude': list of namespaces to NOT capture metrics for. Default: [], none of the namespaces excluded.
# `exclude` takes precedence over `include` in cases when a Namespace is provided under both.
namespaces:
include: []
exclude: []

# Configures the bucket boundaries for all Histogram metrics, the value below is the default.
bucketBoundaries: [0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10, 15, 20, 25, 30]

# Per Metric configuration, allows disabling metrics, dropping labels and change the bucket boundaries.
metricsExposure:
kyverno_policy_execution_duration_seconds:
disabledLabelDimensions: ["resource_kind", "resource_namespace", "resource_request_operation"]
bucketBoundaries: [0.005, 0.01, 0.025]
kyverno_admission_review_duration_seconds:
enabled: false
...
```

`exclude` takes precedence over `include` in cases when a Namespace is provided under both.

## Disabling metrics

Some metrics may generate an excess amount of data which may be undesirable in situations where this incurs additional cost. Some monitoring products and solutions have the ability to selectively disable which metrics are sent to collectors while leaving others enabled.

### Kyverno configuration side
As described [above](#configuring-the-metrics), Kyverno allows disabling metrics, dropping labels and changing the bucket boundaries by changing the `kyverno-metrics` ConfigMap, please refer to the example provided.

### DataDog OpenMetrics side

Disabling select metrics with [DataDog OpenMetrics](https://docs.datadoghq.com/integrations/openmetrics/) can be done by annotating the Kyverno Pod(s) as shown below.

```yaml
Expand Down

0 comments on commit 4ee00e5

Please sign in to comment.