From 31099aaf79b54f7e8c57cf889a126566e5dbde7f Mon Sep 17 00:00:00 2001 From: Jacek Ewertowski Date: Tue, 6 Jun 2023 17:53:53 +0200 Subject: [PATCH] OSSM-3661: Add docs for integration service mesh with monitoring stack Signed-off-by: Jacek Ewertowski [DOC] Document configuring OSSM with OpenShift (or external) Prometheus --- ...grating-with-user-workload-monitoring.adoc | 158 ++++++++++++++++++ service_mesh/v2x/ossm-observability.adoc | 10 ++ 2 files changed, 168 insertions(+) create mode 100644 modules/ossm-integrating-with-user-workload-monitoring.adoc diff --git a/modules/ossm-integrating-with-user-workload-monitoring.adoc b/modules/ossm-integrating-with-user-workload-monitoring.adoc new file mode 100644 index 000000000000..42b991dc3829 --- /dev/null +++ b/modules/ossm-integrating-with-user-workload-monitoring.adoc @@ -0,0 +1,158 @@ +//// +Module included in the following assemblies: +* service_mesh/v2x/ossm-observability.adoc +//// + +:_content-type: PROCEDURE +[id="ossm-integrating-with-user-workload-monitoring_{context}"] += Integrating with user-workload monitoring + +By default, {SMProductName} (OSSM) installs the Service Mesh control plane (SMCP) with a dedicated instance of Prometheus for collecting metrics from a mesh. However, production systems need more advanced monitoring systems, like {product-title} monitoring for user-defined projects. + +The following steps show how to integrate Service Mesh with user-workload monitoring. + +.Prerequisites + +* User-workload monitoring is enabled. +* {SMProductName} Operator 2.4 is installed. + +.Procedure + +. Configure the SMCP for external Prometheus: ++ +[source,yaml] +---- +apiVersion: maistra.io/v2 +kind: ServiceMeshControlPlane +metadata: + name: basic + namespace: istio-system +spec: + addons: + prometheus: + enabled: false # <1> + grafana: + enabled: false # <2> + kiali: + enabled: false +---- +<1> Disable the default Prometheus instance provided by OSSM. +<2> Disable Grafana. It is not supported with an external Prometheus instance. + +. Apply a custom network policy to allow ingress traffic from the monitoring namespace: ++ +[source,yaml] +---- +apiVersion: networking.k8s.io/v1 +kind: NetworkPolicy +metadata: + name: user-workload-access + namespace: bookinfo +spec: + ingress: + - from: + - namespaceSelector: + matchLabels: + network.openshift.io/policy-group: monitoring + podSelector: {} + policyTypes: + - Ingress +---- + +. Apply a `Telemetry` object to enable traffic metrics in Istio proxies: ++ +[source,yaml] +---- +apiVersion: telemetry.istio.io/v1alpha1 +kind: Telemetry +metadata: + name: enable-prometheus-metrics + namespace: istio-system # <1> +spec: + selector: # <2> + matchLabels: + app: bookinfo + metrics: + - providers: + - name: prometheus +---- +<1> A `Telemetry` object created in the control plane namespace applies to all workloads in a mesh. To apply telemetry to only one namespace, create the object in the target namespace. +<2> Optional: Setting the `selector.matchLabels` spec applies the `Telemetry` object to specific workloads in the target namespace. + +. Apply a `ServiceMonitor` object to monitor the Istio control plane: ++ +[source,yaml] +---- +apiVersion: monitoring.coreos.com/v1 +kind: ServiceMonitor +metadata: + name: istiod-monitor + namespace: istio-system # <1> +spec: + targetLabels: + - app + selector: + matchLabels: + istio: pilot + endpoints: + - port: http-monitoring + interval: 30s + relabelings: + - action: replace + replacement: "-" # <2> + targetLabel: mesh_id +---- +<1> Since {product-title} monitoring ignores the `namespaceSelector` spec in `ServiceMonitor` and `PodMonitor` objects, you must apply the `PodMonitor` object in all mesh namespaces, including the control plane namespace. +<2> A shared Prometheus instance can include metrics from multiple meshes. However, if you are deploying Kiali, the metrics must be associated with a single mesh. To associate metrics with a single mesh, add a unique `mesh_id` to each one. Doing so narrows the query scope in Kiali to only the relevant mesh metrics. + +. Apply a `PodMonitor` object to collect metrics from Istio proxies: ++ +[source,yaml] +---- +apiVersion: monitoring.coreos.com/v1 +kind: PodMonitor +metadata: + name: istio-proxies-monitor + namespace: istio-system # <1> +spec: + selector: + matchExpressions: + - key: istio-prometheus-ignore + operator: DoesNotExist + podMetricsEndpoints: + - path: /stats/prometheus + interval: 30s + relabelings: + - action: keep + sourceLabels: [__meta_kubernetes_pod_container_name] + regex: "istio-proxy" + - action: keep + sourceLabels: [__meta_kubernetes_pod_annotationpresent_prometheus_io_scrape] + - action: replace + regex: (\d+);(([A-Fa-f0-9]{1,4}::?){1,7}[A-Fa-f0-9]{1,4}) + replacement: '[$2]:$1' + sourceLabels: [__meta_kubernetes_pod_annotation_prometheus_io_port, + __meta_kubernetes_pod_ip] + targetLabel: __address__ + - action: replace + regex: (\d+);((([0-9]+?)(\.|$)){4}) + replacement: $2:$1 + sourceLabels: [__meta_kubernetes_pod_annotation_prometheus_io_port, + __meta_kubernetes_pod_ip] + targetLabel: __address__ + - action: labeldrop + regex: "__meta_kubernetes_pod_label_(.+)" + - sourceLabels: [__meta_kubernetes_namespace] + action: replace + targetLabel: namespace + - sourceLabels: [__meta_kubernetes_pod_name] + action: replace + targetLabel: pod_name + - action: replace + replacement: "-" # <2> + targetLabel: mesh_id +---- +<1> Since {product-title} monitoring ignores the `namespaceSelector` spec in `ServiceMonitor` and `PodMonitor` objects, you must apply the `PodMonitor` object in all mesh namespaces, including the control plane namespace. +<2> A shared Prometheus instance can include metrics from multiple meshes. However, if you are deploying Kiali, the metrics must be associated with a single mesh. To associate metrics with a single mesh, add a unique `mesh_id` to each one. Doing so narrows the query scope in Kiali to only the relevant mesh metrics. + +. Open the {product-title} web console, and check that metrics are visible. \ No newline at end of file diff --git a/service_mesh/v2x/ossm-observability.adoc b/service_mesh/v2x/ossm-observability.adoc index 785e4dcd9f1f..db83bd6d57ec 100644 --- a/service_mesh/v2x/ossm-observability.adoc +++ b/service_mesh/v2x/ossm-observability.adoc @@ -33,3 +33,13 @@ endif::[] include::modules/ossm-access-grafana.adoc[leveloffset=+1] include::modules/ossm-access-prometheus.adoc[leveloffset=+1] + +include::modules/ossm-integrating-with-user-workload-monitoring.adoc[leveloffset=+1] + +[role="_additional-resources"] +[id="additional-resources_user-workload-monitoring"] +== Additional resources + +ifndef::openshift-rosa,openshift-dedicated[] +* xref:../../monitoring/enabling-monitoring-for-user-defined-projects.adoc[Enabling monitoring for user-defined projects] +endif::[] \ No newline at end of file