[monitoring-docs-restructure] OBSDOCS-978: Document how user-workload monitoring admins can write g…

modules/monitoring-about-creating-alerting-rules-for-user-defined-projects.adoc

@@ -4,7 +4,9 @@
 
 :_mod-docs-content-type: CONCEPT
 [id="about-creating-alerting-rules-for-user-defined-projects_{context}"]
-= About creating alerting rules for user-defined projects
+= Creating alerting rules for user-defined projects
+
+In {product-title}, you can create alerting rules for user-defined projects. Those alerting rules will trigger alerts based on the values of the chosen metrics.
 
 If you create alerting rules for a user-defined project, consider the following key behaviors and important limitations when you define the new rules:
 
@@ -14,6 +16,8 @@ You cannot include metrics from another user-defined project.
 For example, an alerting rule for the `ns1` user-defined project can use metrics exposed by the `ns1` project in addition to core platform metrics, such as CPU and memory metrics.
 However, the rule cannot include metrics from a different `ns2` user-defined project.
 
+* By default, when you create an alerting rule, the `namespace` label is enforced on it even if a rule with the same name exists in another project. To create alerting rules that are not bound to their project of origin, see "Creating cross-project alerting rules for user-defined projects".
+
 * To reduce latency and to minimize the load on core platform monitoring components, you can add the `openshift.io/prometheus-rule-evaluation-scope: leaf-prometheus` label to a rule.
 This label forces only the Prometheus instance deployed in the `openshift-user-workload-monitoring` project to evaluate the alerting rule and prevents the Thanos Ruler instance from doing so.
 +
@@ -22,3 +26,6 @@ This label forces only the Prometheus instance deployed in the `openshift-user-w
 If an alerting rule has this label, your alerting rule can use only those metrics exposed by your user-defined project.
 Alerting rules you create based on default platform metrics might not trigger alerts.
 ====
+
+
+

modules/monitoring-creating-alerting-rules-for-user-defined-projects.adoc

@@ -10,9 +10,7 @@ You can create alerting rules for user-defined projects. Those alerting rules wi
 
 [NOTE]
 ====
-* When you create an alerting rule, a project label is enforced on it even if a rule with the same name exists in another project.
-
-* To help users understand the impact and cause of the alert, ensure that your alerting rule contains an alert message and severity value.
+To help users understand the impact and cause of the alert, ensure that your alerting rule contains an alert message and severity value.
 ====
 
 .Prerequisites

modules/monitoring-creating-cross-project-alerting-rules-for-user-defined-projects.adoc

@@ -0,0 +1,99 @@
+// Module included in the following assemblies:
+//
+// * observability/monitoring/managing-alerts.adoc
+
+:_mod-docs-content-type: PROCEDURE
+[id="creating-cross-project-alerting-rules-for-user-defined-projects_{context}"]
+= Creating cross-project alerting rules for user-defined projects
+
+You can create alerting rules for user-defined projects that are not bound to their project of origin by configuring a project in the `user-workload-monitoring-config` config map. This allows you to create generic alerting rules that get applied to multiple user-defined projects instead of having individual `PrometheusRule` objects in each user project.
+
+.Prerequisites
+
+ifndef::openshift-dedicated,openshift-rosa[]
+* If you are a cluster administrator, you have access to the cluster as a user with the `cluster-admin` cluster role.
+* If you are a non-administrator user, you have access to the cluster as a user with the following user roles:
+** The `user-workload-monitoring-config-edit` role in the `openshift-user-workload-monitoring` project to edit the `user-workload-monitoring-config` config map.
+** The `monitoring-rules-edit` cluster role for the project where you want to create an alerting rule.
+* A cluster administrator has enabled monitoring for user-defined projects.
+endif::openshift-dedicated,openshift-rosa[]
+ifdef::openshift-dedicated,openshift-rosa[]
+* You have access to the cluster as a user with the `dedicated-admin` role.
++
+[NOTE]
+====
+If you are a non-administrator user, you can still create cross-project alerting rules if you have the `monitoring-rules-edit` cluster role for the project where you want to create an alerting rule. However, that project needs to be configured in the `user-workload-monitoring-config` config map under the `namespacesWithoutLabelEnforcement` property, which can be done only by cluster administrators.
+====
+* The `user-workload-monitoring-config` `ConfigMap` object exists. This object is created by default when the cluster is created.
+endif::openshift-dedicated,openshift-rosa[]
+* You have installed the OpenShift CLI (`oc`).
+
+.Procedure
+
+. Edit the `user-workload-monitoring-config` config map in the `openshift-user-workload-monitoring` project:
++
+[source,terminal]
+----
+$ oc -n openshift-user-workload-monitoring edit configmap user-workload-monitoring-config
+----
+
+. Configure projects in which you want to create alerting rules that are not bound to a specific project:
++
+[source,yaml]
+----
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: user-workload-monitoring-config
+  namespace: openshift-user-workload-monitoring
+data:
+  config.yaml: |
+    namespacesWithoutLabelEnforcement: [ <namespace> ] # <1>
+    # ...
+----
+<1> Specify one or more projects in which you want to create cross-project alerting rules. Prometheus and Thanos Ruler for user-defined monitoring do not enforce the `namespace` label in `PrometheusRule` objects created in these projects.
+
+. Create a YAML file for alerting rules. In this example, it is called `example-cross-project-alerting-rule.yaml`.
+
+. Add an alerting rule configuration to the YAML file.
+The following example creates a new cross-project alerting rule called `example-security`. The alerting rule fires when a user project does not enforce the restricted pod security policy:
++
+.Example cross-project alerting rule
+[source,yaml]
+----
+apiVersion: monitoring.coreos.com/v1
+kind: PrometheusRule
+metadata:
+  name: example-security
+namespace: ns1 #<1>
+spec:
+  groups:
+    - name: pod-security-policy
+      rules:
+        - alert: "ProjectNotEnforcingRestrictedPolicy" # <2>
+          for: 5m # <3>
+          expr: kube_namespace_labels{namespace!~"(openshift|kube).*|default",label_pod_security_kubernetes_io_enforce!="restricted"} # <4>
+          annotations:
+            message: "Restricted policy not enforced. Project {{ $labels.namespace }} does not enforce the restricted pod security policy." #<5>
+          labels:
+            severity: warning # <6>
+----
+<1> Ensure that you specify the project that you defined in the `namespacesWithoutLabelEnforcement` field.
+<2> The name of the alerting rule you want to create.
+<3> The duration for which the condition should be true before an alert is fired.
+<4> The PromQL query expression that defines the new rule.
+<5> The message associated with the alert.
+<6> The severity that alerting rule assigns to the alert.
++
+[IMPORTANT]
+====
+Ensure that you create a specific cross-project alerting rule in only one of the projects that you specified in the `namespacesWithoutLabelEnforcement` field.
+If you create the same cross-project alerting rule in multiple projects, it results in repeated alerts.
+====
+
+. Apply the configuration file to the cluster:
++
+[source,terminal]
+----
+$ oc apply -f example-cross-project-alerting-rule.yaml
+----

modules/monitoring-disabling-cross-project-alerting-rules-for-user-defined-projects.adoc

@@ -0,0 +1,52 @@
+// Module included in the following assemblies:
+//
+// * observability/monitoring/managing-alerts.adoc
+
+:_mod-docs-content-type: PROCEDURE
+[id="disabling-cross-project-alerting-rules-for-user-defined-projects_{context}"]
+= Disabling cross-project alerting rules for user-defined projects
+
+Creating cross-project alerting rules for user-defined projects is enabled by default. Cluster administrators can disable the capability in the `cluster-monitoring-config` config map for the following reasons:
+
+* To prevent user-defined monitoring from overloading the cluster monitoring stack. 
+* To prevent buggy alerting rules from being applied to the cluster without having to identify the rule that causes the issue.
+
+.Prerequisites
+
+ifndef::openshift-dedicated,openshift-rosa[]
+* You have access to the cluster as a user with the `cluster-admin` cluster role.
+endif::openshift-dedicated,openshift-rosa[]
+ifdef::openshift-dedicated,openshift-rosa[]
+* You have access to the cluster as a user with the `dedicated-admin` role.
+endif::openshift-dedicated,openshift-rosa[]
+* You have installed the OpenShift CLI (`oc`).
+
+.Procedure
+
+. Edit the `cluster-monitoring-config` config map in the `openshift-monitoring` project:
++
+[source,terminal]
+----
+$ oc -n openshift-monitoring edit configmap cluster-monitoring-config
+----
+
+. In the `cluster-monitoring-config` config map, disable the option to create cross-project alerting rules by setting the `rulesWithoutLabelEnforcementAllowed` value under `data/config.yaml/userWorkload` to `false`:
++
+[source,yaml]
+----
+kind: ConfigMap
+apiVersion: v1
+metadata:
+  name: cluster-monitoring-config
+  namespace: openshift-monitoring
+data:
+  config.yaml: |
+    userWorkload:
+      rulesWithoutLabelEnforcementAllowed: false
+    # ...
+----
+
+. Save the file to apply the changes.
+
+
+

modules/monitoring-managing-alerting-rules-for-user-defined-projects.adoc

@@ -9,7 +9,7 @@
 
 {product-title} monitoring ships with a set of default alerting rules. As a cluster administrator, you can view the default alerting rules.
 
-In {product-title} {product-version}, you can create, view, edit, and remove alerting rules in user-defined projects.
+In {product-title}, you can view, edit, and remove alerting rules in user-defined projects.
 
 ifdef::openshift-rosa,openshift-dedicated[]
 [IMPORTANT]

observability/monitoring/managing-alerts.adoc

@@ -56,33 +56,28 @@ include::modules/monitoring-modifying-core-platform-alerting-rules.adoc[leveloff
 * See the link:https://prometheus.io/docs/practices/alerting/[Prometheus alerting documentation] for further guidelines on optimizing alerts.
 endif::openshift-dedicated,openshift-rosa[]
 
-// Managing alerting rules for user-defined projects
-include::modules/monitoring-managing-alerting-rules-for-user-defined-projects.adoc[leveloffset=+1]
+// Creating alerting rules for user-defined projects
+include::modules/monitoring-about-creating-alerting-rules-for-user-defined-projects.adoc[leveloffset=+1]
 include::modules/monitoring-optimizing-alerting-for-user-defined-projects.adoc[leveloffset=+2]
-
-[role="_additional-resources"]
-.Additional resources
-* See the link:https://prometheus.io/docs/practices/alerting/[Prometheus alerting documentation] for further guidelines on optimizing alerts
-ifndef::openshift-rosa,openshift-dedicated[]
-* See xref:../../observability/monitoring/monitoring-overview.adoc#monitoring-overview[Monitoring overview] for details about {product-title} {product-version} monitoring architecture
-endif::[]
-
-// creating alerting rules for user defined projects
-include::modules/monitoring-about-creating-alerting-rules-for-user-defined-projects.adoc[leveloffset=+2]
 include::modules/monitoring-creating-alerting-rules-for-user-defined-projects.adoc[leveloffset=+2]
+include::modules/monitoring-creating-cross-project-alerting-rules-for-user-defined-projects.adoc[leveloffset=+2]
 
 [role="_additional-resources"]
 .Additional resources
-* See xref:../../observability/monitoring/monitoring-overview.adoc#monitoring-overview[Monitoring overview] for details about {product-title} {product-version} monitoring architecture.
+* link:https://prometheus.io/docs/practices/alerting/[Prometheus alerting documentation]
+* xref:../../observability/monitoring/monitoring-overview.adoc#monitoring-overview[Monitoring overview]
 
+// Managing alerting rules for user-defined projects
+include::modules/monitoring-managing-alerting-rules-for-user-defined-projects.adoc[leveloffset=+1]
 include::modules/monitoring-accessing-alerting-rules-for-your-project.adoc[leveloffset=+2]
 include::modules/monitoring-listing-alerting-rules-for-all-projects-in-a-single-view.adoc[leveloffset=+2]
 include::modules/monitoring-removing-alerting-rules-for-user-defined-projects.adoc[leveloffset=+2]
+include::modules/monitoring-disabling-cross-project-alerting-rules-for-user-defined-projects.adoc[leveloffset=+2]
 
 [role="_additional-resources"]
 .Additional resources
 
-* See the link:https://prometheus.io/docs/alerting/alertmanager/[Alertmanager documentation]
+* link:https://prometheus.io/docs/alerting/alertmanager/[Alertmanager documentation]
 
 // Sending notifications to external systems
 include::modules/monitoring-sending-notifications-to-external-systems.adoc[leveloffset=+1]