Merge pull request #61854 from gwynnemonahan/OSSM-3661

OSSM-3661: [DOC] Document configuring OSSM with OpenShift (or external) Prometheus

modules/ossm-integrating-with-user-workload-monitoring.adoc

@@ -0,0 +1,158 @@
+////
+Module included in the following assemblies:
+* service_mesh/v2x/ossm-observability.adoc
+////
+
+:_content-type: PROCEDURE
+[id="ossm-integrating-with-user-workload-monitoring_{context}"]
+= Integrating with user-workload monitoring
+
+By default, {SMProductName} (OSSM) installs the Service Mesh control plane (SMCP) with a dedicated instance of Prometheus for collecting metrics from a mesh. However, production systems need more advanced monitoring systems, like {product-title} monitoring for user-defined projects.
+
+The following steps show how to integrate Service Mesh with user-workload monitoring.
+
+.Prerequisites
+
+* User-workload monitoring is enabled.
+* {SMProductName} Operator 2.4 is installed.
+
+.Procedure
+
+. Configure the SMCP for external Prometheus:
++
+[source,yaml]
+----
+apiVersion: maistra.io/v2
+kind: ServiceMeshControlPlane
+metadata:
+  name: basic
+  namespace: istio-system
+spec:
+  addons:
+    prometheus:
+      enabled: false # <1>
+    grafana:
+      enabled: false # <2>
+    kiali:
+      enabled: false
+----
+<1> Disable the default Prometheus instance provided by OSSM.
+<2> Disable Grafana. It is not supported with an external Prometheus instance.
+
+. Apply a custom network policy to allow ingress traffic from the monitoring namespace:
++
+[source,yaml]
+----
+apiVersion: networking.k8s.io/v1
+kind: NetworkPolicy
+metadata:
+  name: user-workload-access
+  namespace: bookinfo
+spec:
+  ingress:
+  - from:
+    - namespaceSelector:
+        matchLabels:
+          network.openshift.io/policy-group: monitoring
+  podSelector: {}
+  policyTypes:
+  - Ingress
+----
+
+. Apply a `Telemetry` object to enable traffic metrics in Istio proxies:
++
+[source,yaml]
+----
+apiVersion: telemetry.istio.io/v1alpha1
+kind: Telemetry
+metadata:
+  name: enable-prometheus-metrics
+  namespace: istio-system # <1>
+spec:
+  selector: # <2>
+    matchLabels:
+      app: bookinfo 
+  metrics:
+  - providers:
+    - name: prometheus
+----
+<1> A `Telemetry` object created in the control plane namespace applies to all workloads in a mesh. To apply telemetry to only one namespace, create the object in the target namespace.
+<2> Optional: Setting the `selector.matchLabels` spec applies the `Telemetry` object to specific workloads in the target namespace.
+
+. Apply a `ServiceMonitor` object to monitor the Istio control plane:
++
+[source,yaml]
+----
+apiVersion: monitoring.coreos.com/v1
+kind: ServiceMonitor
+metadata:
+  name: istiod-monitor
+  namespace: istio-system # <1>
+spec:
+  targetLabels:
+  - app
+  selector:
+    matchLabels:
+      istio: pilot
+  endpoints:
+  - port: http-monitoring
+    interval: 30s
+    relabelings:
+    - action: replace
+      replacement: "<smcp_name>-<smcp_namespace>" # <2>
+      targetLabel: mesh_id
+----
+<1> Since {product-title} monitoring ignores the `namespaceSelector` spec in `ServiceMonitor` and `PodMonitor` objects, you must apply the `PodMonitor` object in all mesh namespaces, including the control plane namespace.
+<2> A shared Prometheus instance can include metrics from multiple meshes. However, if you are deploying Kiali, the metrics must be associated with a single mesh. To associate metrics with a single mesh, add a unique `mesh_id` to each one. Doing so narrows the query scope in Kiali to only the relevant mesh metrics.
+
+. Apply a `PodMonitor` object to collect metrics from Istio proxies:
++
+[source,yaml]
+----
+apiVersion: monitoring.coreos.com/v1
+kind: PodMonitor
+metadata:
+  name: istio-proxies-monitor
+  namespace: istio-system # <1>
+spec:
+  selector:
+    matchExpressions:
+    - key: istio-prometheus-ignore
+      operator: DoesNotExist
+  podMetricsEndpoints:
+  - path: /stats/prometheus
+    interval: 30s
+    relabelings:
+    - action: keep
+      sourceLabels: [__meta_kubernetes_pod_container_name]
+      regex: "istio-proxy"
+    - action: keep
+      sourceLabels: [__meta_kubernetes_pod_annotationpresent_prometheus_io_scrape]
+    - action: replace
+      regex: (\d+);(([A-Fa-f0-9]{1,4}::?){1,7}[A-Fa-f0-9]{1,4})
+      replacement: '[$2]:$1'
+      sourceLabels: [__meta_kubernetes_pod_annotation_prometheus_io_port, 
+      __meta_kubernetes_pod_ip]
+      targetLabel: __address__
+    - action: replace
+      regex: (\d+);((([0-9]+?)(\.|$)){4})
+      replacement: $2:$1
+      sourceLabels: [__meta_kubernetes_pod_annotation_prometheus_io_port, 
+      __meta_kubernetes_pod_ip]
+      targetLabel: __address__
+    - action: labeldrop
+      regex: "__meta_kubernetes_pod_label_(.+)"
+    - sourceLabels: [__meta_kubernetes_namespace]
+      action: replace
+      targetLabel: namespace
+    - sourceLabels: [__meta_kubernetes_pod_name]
+      action: replace
+      targetLabel: pod_name
+    - action: replace
+      replacement: "<smcp_name>-<smcp_namespace>" # <2>
+      targetLabel: mesh_id
+----
+<1> Since {product-title} monitoring ignores the `namespaceSelector` spec in `ServiceMonitor` and `PodMonitor` objects, you must apply the `PodMonitor` object in all mesh namespaces, including the control plane namespace.
+<2> A shared Prometheus instance can include metrics from multiple meshes. However, if you are deploying Kiali, the metrics must be associated with a single mesh. To associate metrics with a single mesh, add a unique `mesh_id` to each one. Doing so narrows the query scope in Kiali to only the relevant mesh metrics.
+
+. Open the {product-title} web console, and check that metrics are visible.

service_mesh/v2x/ossm-observability.adoc

@@ -33,3 +33,13 @@ endif::[]
 include::modules/ossm-access-grafana.adoc[leveloffset=+1]
 
 include::modules/ossm-access-prometheus.adoc[leveloffset=+1]
+
+include::modules/ossm-integrating-with-user-workload-monitoring.adoc[leveloffset=+1]
+
+[role="_additional-resources"]
+[id="additional-resources_user-workload-monitoring"]
+== Additional resources 
+
+ifndef::openshift-rosa,openshift-dedicated[]
+* xref:../../monitoring/enabling-monitoring-for-user-defined-projects.adoc[Enabling monitoring for user-defined projects]
+endif::[]