diff --git a/modules/monitoring-about-creating-alerting-rules-for-user-defined-projects.adoc b/modules/monitoring-about-creating-alerting-rules-for-user-defined-projects.adoc new file mode 100644 index 000000000000..de1f3c1360dc --- /dev/null +++ b/modules/monitoring-about-creating-alerting-rules-for-user-defined-projects.adoc @@ -0,0 +1,24 @@ +// Module included in the following assemblies: +// +// * monitoring/managing-alerts.adoc + +:_mod-docs-content-type: CONCEPT +[id="about-creating-alerting-rules-for-user-defined-projects_{context}"] += About creating alerting rules for user-defined projects + +If you create alerting rules for a user-defined project, consider the following key behaviors and important limitations when you define the new rules: + +* A user-defined alerting rule can include metrics exposed by its own project in addition to the default metrics from core platform monitoring. +You cannot include metrics from another user-defined project. ++ +For example, an alerting rule for the `ns1` user-defined project can use metrics exposed by the `ns1` project in addition to core platform metrics, such as CPU and memory metrics. +However, the rule cannot include metrics from a different `ns2` user-defined project. + +* To reduce latency and to minimize the load on core platform monitoring components, you can add the `openshift.io/prometheus-rule-evaluation-scope: leaf-prometheus` label to a rule. +This label forces only the Prometheus instance deployed in the `openshift-user-workload-monitoring` project to evaluate the alerting rule and prevents the Thanos Ruler instance from doing so. ++ +[IMPORTANT] +==== +If an alerting rule has this label, your alerting rule can use only those metrics exposed by your user-defined project. +Alerting rules you create based on default platform metrics might not trigger alerts. +==== diff --git a/modules/monitoring-creating-alerting-rules-for-user-defined-projects.adoc b/modules/monitoring-creating-alerting-rules-for-user-defined-projects.adoc index bb2ee4187ab4..8e13e87e2e23 100644 --- a/modules/monitoring-creating-alerting-rules-for-user-defined-projects.adoc +++ b/modules/monitoring-creating-alerting-rules-for-user-defined-projects.adoc @@ -6,7 +6,7 @@ [id="creating-alerting-rules-for-user-defined-projects_{context}"] = Creating alerting rules for user-defined projects -You can create alerting rules for user-defined projects. Those alerting rules will fire alerts based on the values of chosen metrics. +You can create alerting rules for user-defined projects. Those alerting rules will trigger alerts based on the values of the chosen metrics. .Prerequisites @@ -41,15 +41,6 @@ spec: ---- + This configuration creates an alerting rule named `example-alert`. The alerting rule fires an alert when the `version` metric exposed by the sample service becomes `0`. -+ -[IMPORTANT] -==== -A user-defined alerting rule can include metrics for its own project and cluster metrics. You cannot include metrics for another user-defined project. - -For example, an alerting rule for the user-defined project `ns1` can have metrics from `ns1` and cluster metrics, such as the CPU and memory metrics. However, the rule cannot include metrics from `ns2`. - -Additionally, you cannot create alerting rules for the `openshift-*` core {product-title} projects. {product-title} monitoring by default provides a set of alerting rules for these projects. -==== . Apply the configuration file to the cluster: + @@ -57,5 +48,3 @@ Additionally, you cannot create alerting rules for the `openshift-*` core {produ ---- $ oc apply -f example-app-alerting-rule.yaml ---- -+ -It takes some time to create the alerting rule. diff --git a/modules/monitoring-optimizing-alerting-for-user-defined-projects.adoc b/modules/monitoring-optimizing-alerting-for-user-defined-projects.adoc index f3c351da525c..f4626df8e13e 100644 --- a/modules/monitoring-optimizing-alerting-for-user-defined-projects.adoc +++ b/modules/monitoring-optimizing-alerting-for-user-defined-projects.adoc @@ -17,10 +17,3 @@ You can optimize alerting for your own projects by considering the following rec * *Provide clear alert messaging*. State the symptom and recommended actions in the alert message. * *Include severity levels in your alerting rules*. The severity of an alert depends on how you need to react if the reported symptom occurs. For example, a critical alert should be triggered if a symptom requires immediate attention by an individual or a critical response team. - -* *Optimize alert routing*. Deploy an alerting rule directly on the Prometheus instance in the `openshift-user-workload-monitoring` project if the rule does not query default {product-title} metrics. This reduces latency for alerting rules and minimizes the load on monitoring components. -+ -[WARNING] -==== -Default {product-title} metrics for user-defined projects provide information about CPU and memory usage, bandwidth statistics, and packet rate information. Those metrics cannot be included in an alerting rule if you route the rule directly to the Prometheus instance in the `openshift-user-workload-monitoring` project. Alerting rule optimization should be used only if you have read the documentation and have a comprehensive understanding of the monitoring architecture. -==== diff --git a/modules/monitoring-reducing-latency-for-alerting-rules-that-do-not-query-platform-metrics.adoc b/modules/monitoring-reducing-latency-for-alerting-rules-that-do-not-query-platform-metrics.adoc deleted file mode 100644 index c5feeaa69eae..000000000000 --- a/modules/monitoring-reducing-latency-for-alerting-rules-that-do-not-query-platform-metrics.adoc +++ /dev/null @@ -1,54 +0,0 @@ -// Module included in the following assemblies: -// -// * monitoring/managing-alerts.adoc - -:_mod-docs-content-type: PROCEDURE -[id="reducing-latency-for-alerting-rules-that-do-not-query-platform-metrics_{context}"] -= Reducing latency for alerting rules that do not query platform metrics - -If an alerting rule for a user-defined project does not query default cluster metrics, you can deploy the rule directly on the Prometheus instance in the `openshift-user-workload-monitoring` project. This reduces latency for alerting rules by bypassing Thanos Ruler when it is not required. This also helps to minimize the overall load on monitoring components. - -[WARNING] -==== -Default {product-title} metrics for user-defined projects provide information about CPU and memory usage, bandwidth statistics, and packet rate information. Those metrics cannot be included in an alerting rule if you deploy the rule directly to the Prometheus instance in the `openshift-user-workload-monitoring` project. The procedure outlined in this section should only be used if you have read the documentation and have a comprehensive understanding of the monitoring architecture. -==== - -.Prerequisites - -* You have enabled monitoring for user-defined projects. -* You are logged in as a user that has the `monitoring-rules-edit` cluster role for the project where you want to create an alerting rule. -* You have installed the OpenShift CLI (`oc`). - -.Procedure - -. Create a YAML file for alerting rules. In this example, it is called `example-app-alerting-rule.yaml`. - -. Add an alerting rule configuration to the YAML file that includes a label with the key `openshift.io/prometheus-rule-evaluation-scope` and value `leaf-prometheus`. For example: -+ -[source,yaml] ----- -apiVersion: monitoring.coreos.com/v1 -kind: PrometheusRule -metadata: - name: example-alert - namespace: ns1 - labels: - openshift.io/prometheus-rule-evaluation-scope: leaf-prometheus -spec: - groups: - - name: example - rules: - - alert: VersionAlert - expr: version{job="prometheus-example-app"} == 0 ----- -+ -If that label is present, the alerting rule is deployed on the Prometheus instance in the `openshift-user-workload-monitoring` project. If the label is not present, the alerting rule is deployed to Thanos Ruler. - -. Apply the configuration file to the cluster: -+ -[source,terminal] ----- -$ oc apply -f example-app-alerting-rule.yaml ----- -+ -It takes some time to create the alerting rule. diff --git a/monitoring/managing-alerts.adoc b/monitoring/managing-alerts.adoc index 7da8f0d60386..084f171b5b19 100644 --- a/monitoring/managing-alerts.adoc +++ b/monitoring/managing-alerts.adoc @@ -53,8 +53,9 @@ ifndef::openshift-rosa,openshift-dedicated[] * See xref:../monitoring/monitoring-overview.adoc#monitoring-overview[Monitoring overview] for details about {product-title} {product-version} monitoring architecture endif::[] +// creating alerting rules for user defined projects +include::modules/monitoring-about-creating-alerting-rules-for-user-defined-projects.adoc[leveloffset=+2] include::modules/monitoring-creating-alerting-rules-for-user-defined-projects.adoc[leveloffset=+2] -include::modules/monitoring-reducing-latency-for-alerting-rules-that-do-not-query-platform-metrics.adoc[leveloffset=+2] [role="_additional-resources"] .Additional resources