From 7f2991a7f4b20c317a8e0fa976f835074885ae90 Mon Sep 17 00:00:00 2001 From: Brian Burt Date: Thu, 11 Aug 2022 16:37:05 -0400 Subject: [PATCH] RHDEVDOCS-4036 - CMO config map options for body size limits for scraped metrics --- ...-body-size-limit-for-metrics-scraping.adoc | 61 +++++++++++++++++++ .../configuring-the-monitoring-stack.adoc | 10 ++- 2 files changed, 70 insertions(+), 1 deletion(-) create mode 100644 modules/monitoring-setting-the-body-size-limit-for-metrics-scraping.adoc diff --git a/modules/monitoring-setting-the-body-size-limit-for-metrics-scraping.adoc b/modules/monitoring-setting-the-body-size-limit-for-metrics-scraping.adoc new file mode 100644 index 000000000000..b15cdb3d1ce9 --- /dev/null +++ b/modules/monitoring-setting-the-body-size-limit-for-metrics-scraping.adoc @@ -0,0 +1,61 @@ +// Module included in the following assemblies: +// +// * monitoring/configuring-the-monitoring-stack.adoc + +:_content-type: PROCEDURE +[id="setting-the-body-size-limit-for-metrics-scraping_{context}"] += Setting the body size limit for metrics scraping + +By default, no limit exists for the uncompressed body size for data returned from scraped metrics targets. +You can set a body size limit to help avoid situations in which Prometheus consumes excessive amounts of memory when scraped targets return a response that contains a large amount of data. +In addition, by setting a body size limit, you can reduce the impact that a malicious target might have on Prometheus and on the cluster as a whole. + +After you set a value for `enforcedBodySizeLimit`, the alert `PrometheusScrapeBodySizeLimitHit` fires when at least one Prometheus scrape target replies with a response body larger than the configured value. + +[NOTE] +==== +If metrics data scraped from a target has an uncompressed body size exceeding the configured size limit, the scrape fails. +Prometheus then considers this target to be down and sets its `up` metric value to `0`, which can trigger the `TargetDown` alert. +==== + +.Prerequisites + +* You have access to the cluster as a user with the `cluster-admin` role. +* You have installed the OpenShift CLI (`oc`). + +.Procedure + +. Edit the `cluster-monitoring-config` `ConfigMap` object in the `openshift-monitoring` namespace: ++ +[source,terminal] +---- +$ oc -n openshift-monitoring edit configmap cluster-monitoring-config +---- + +. Add a value for `enforcedBodySizeLimit` to `data/config.yaml/prometheusK8s` to limit the body size that can be accepted per target scrape: ++ +[source,yaml] +---- +apiVersion: v1 +kind: ConfigMap +metadata: + name: cluster-monitoring-config + namespace: openshift-monitoring +data: + config.yaml: |- + prometheusK8s: + enforcedBodySizeLimit: 40MB <1> +---- +<1> Specify the maximum body size for scraped metrics targets. +This `enforcedBodySizeLimit` example limits the uncompressed size per target scrape to 40 megabytes. +Valid numeric values use the Prometheus data size format: B (bytes), KB (kilobytes), MB (megabytes), GB (gigabytes), TB (terabytes), PB (petabytes), and EB (exabytes). +The default value is `0`, which specifies no limit. +You can also set the value to `automatic` to calculate the limit automatically based on cluster capacity. + +. Save the file to apply the changes automatically. ++ +[WARNING] +==== +When you save changes to a `cluster-monitoring-config` config map, the pods and other resources in the `openshift-monitoring` project might be redeployed. +The running monitoring processes in that project might also restart. +==== diff --git a/monitoring/configuring-the-monitoring-stack.adoc b/monitoring/configuring-the-monitoring-stack.adoc index a49966a09171..e31e24e93ed5 100644 --- a/monitoring/configuring-the-monitoring-stack.adoc +++ b/monitoring/configuring-the-monitoring-stack.adoc @@ -68,6 +68,14 @@ include::modules/monitoring-assigning-tolerations-to-monitoring-components.adoc[ * See the xref:../nodes/scheduling/nodes-scheduler-taints-tolerations.adoc#nodes-scheduler-taints-tolerations[{product-title} documentation] on taints and tolerations * See the link:https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/[Kubernetes documentation] on taints and tolerations +// Setting the body size limit for metrics scraping +include::modules/monitoring-setting-the-body-size-limit-for-metrics-scraping.adoc[leveloffset=+1] + +[role="_additional-resources"] +.Additional resources + +* link:https://prometheus.io/docs/prometheus/latest/configuration/configuration/#scrape_config[Prometheus scrape configuration documentation] + // Configuring persistent storage include::modules/monitoring-configuring-persistent-storage.adoc[leveloffset=+1] include::modules/monitoring-configuring-a-local-persistent-volume-claim.adoc[leveloffset=+2] @@ -168,4 +176,4 @@ include::modules/monitoring-disabling-the-local-alertmanager.adoc[leveloffset=+1 == Next steps * xref:../monitoring/enabling-monitoring-for-user-defined-projects.adoc#enabling-monitoring-for-user-defined-projects[Enabling monitoring for user-defined projects] -* Learn about xref:../support/remote_health_monitoring/opting-out-of-remote-health-reporting.adoc#opting-out-remote-health-reporting_opting-out-remote-health-reporting[remote health reporting] and, if necessary, opt out of it. +* Learn about xref:../support/remote_health_monitoring/opting-out-of-remote-health-reporting.adoc#opting-out-remote-health-reporting_opting-out-remote-health-reporting[remote health reporting] and, if necessary, opt out of it. \ No newline at end of file