From e87c3f42321d394446be71ad1824c6860e8b597b Mon Sep 17 00:00:00 2001 From: Brian Burt Date: Wed, 15 Sep 2021 14:39:51 +0100 Subject: [PATCH] monitoring: Describe query log file feature for Prometheus --- ...setting-query-log-file-for-prometheus.adoc | 132 ++++++++++++++++++ .../configuring-the-monitoring-stack.adoc | 5 +- 2 files changed, 136 insertions(+), 1 deletion(-) create mode 100644 modules/monitoring-setting-query-log-file-for-prometheus.adoc diff --git a/modules/monitoring-setting-query-log-file-for-prometheus.adoc b/modules/monitoring-setting-query-log-file-for-prometheus.adoc new file mode 100644 index 000000000000..3b8bcb1fe5e3 --- /dev/null +++ b/modules/monitoring-setting-query-log-file-for-prometheus.adoc @@ -0,0 +1,132 @@ +// Module included in the following assemblies: +// +// * monitoring/configuring-the-monitoring-stack.adoc + +[id="setting-query-log-file-for-prometheus_{context}"] +:_content-type: PROCEDURE += Enabling the query log file for Prometheus + +[role="_abstract"] +You can configure Prometheus to write all queries that have been run by the engine to a log file. You can do so for default platform monitoring and for user-defined workload monitoring. + +[IMPORTANT] +==== +Because log rotation is not supported, only enable this feature temporarily when you need to troubleshoot an issue. After you finish troubleshooting, disable query logging by reverting the changes you made to the `ConfigMap` object to enable the feature. +==== + +.Prerequisites + +* You have installed the OpenShift CLI (`oc`). +* *If you are enabling the query log file feature for Prometheus in the `openshift-monitoring` project*: +** You have access to the cluster as a user with the `cluster-admin` role. +** You have created the `cluster-monitoring-config` `ConfigMap` object. +* *If you are enabling the query log file feature for Prometheus in the `openshift-user-workload-monitoring` project*: +** You have access to the cluster as a user with the `cluster-admin` role, or as a user with the `user-workload-monitoring-config-edit` role in the `openshift-user-workload-monitoring` project. +** You have created the `user-workload-monitoring-config` `ConfigMap` object. + +.Procedure + +** *To set the query log file for Prometheus in the `openshift-monitoring` project*: +. Edit the `cluster-monitoring-config` `ConfigMap` object in the `openshift-monitoring` project: ++ +[source,terminal] +---- +$ oc -n openshift-monitoring edit configmap cluster-monitoring-config +---- ++ +. Add `queryLogFile: ` for `prometheusK8s` under `data/config.yaml`: ++ +[source,yaml] +---- +apiVersion: v1 +kind: ConfigMap +metadata: + name: cluster-monitoring-config + namespace: openshift-monitoring +data: + config.yaml: | + prometheusK8s: + queryLogFile: <1> +---- +<1> The full path to the file in which queries will be logged. ++ +. Save the file to apply the changes. ++ +[WARNING] +==== +When you save changes to a monitoring config map, pods and other resources in the related project might be redeployed. The running monitoring processes in that project might also be restarted. +==== ++ +. Verify that the pods for the component are running. The following sample command lists the status of pods in the `openshift-monitoring` project: ++ +[source,terminal] +---- +$ oc -n openshift-monitoring get pods +---- ++ +. Read the query log: ++ +[source,terminal] +---- +$ oc -n openshift-monitoring exec prometheus-k8s-0 -- cat +---- ++ +[IMPORTANT] +==== +Revert the setting in the config map after you have examined the logged query information. +==== + +** *To set the query log file for Prometheus in the `openshift-user-workload-monitoring` project*: +. Edit the `user-workload-monitoring-config` `ConfigMap` object in the `openshift-user-workload-monitoring` project: ++ +[source,terminal] +---- +$ oc -n openshift-user-workload-monitoring edit configmap user-workload-monitoring-config +---- ++ +. Add `queryLogFile: ` for `prometheus` under `data/config.yaml`: ++ +[source,yaml] +---- +apiVersion: v1 +kind: ConfigMap +metadata: + name: user-workload-monitoring-config + namespace: openshift-user-workload-monitoring +data: + config.yaml: | + prometheus: + queryLogFile: <1> +---- +<1> The full path to the file in which queries will be logged. ++ +. Save the file to apply the changes. ++ +[NOTE] +==== +Configurations applied to the `user-workload-monitoring-config` `ConfigMap` object are not activated unless a cluster administrator has enabled monitoring for user-defined projects. +==== ++ +[WARNING] +==== +When you save changes to a monitoring config map, pods and other resources in the related project might be redeployed. The running monitoring processes in that project might also be restarted. +==== ++ +. Verify that the pods for the component are running. The following example command lists the status of pods in the `openshift-user-workload-monitoring` project: ++ +[source,terminal] +---- +$ oc -n openshift-user-workload-monitoring get pods +---- ++ +. Read the query log: ++ +[source,terminal] +---- +$ oc -n openshift-user-workload-monitoring exec prometheus-user-workload-0 -- cat +---- ++ +[IMPORTANT] +==== +Revert the setting in the config map after you have examined the logged query information. +==== diff --git a/monitoring/configuring-the-monitoring-stack.adoc b/monitoring/configuring-the-monitoring-stack.adoc index 5663b63f7dcb..97512cade972 100644 --- a/monitoring/configuring-the-monitoring-stack.adoc +++ b/monitoring/configuring-the-monitoring-stack.adoc @@ -149,11 +149,14 @@ include::modules/monitoring-attaching-additional-labels-to-your-time-series-and- // Setting log levels for monitoring components include::modules/monitoring-setting-log-levels-for-monitoring-components.adoc[leveloffset=+1] +// Setting query log for Prometheus +include::modules/monitoring-setting-query-log-file-for-prometheus.adoc[leveloffset=+1] + [role="_additional-resources"] .Additional resources * See xref:../monitoring/configuring-the-monitoring-stack.adoc#preparing-to-configure-the-monitoring-stack[Preparing to configure the monitoring stack] for steps to create monitoring config maps -* xref:../monitoring/enabling-monitoring-for-user-defined-projects.adoc#enabling-monitoring-for-user-defined-projects[Enabling monitoring for user-defined projects] +* See xref:../monitoring/enabling-monitoring-for-user-defined-projects.adoc#enabling-monitoring-for-user-defined-projects[Enabling monitoring for user-defined projects] for steps to enable user-defined monitoring. // Disabling the default Grafana deployment include::modules/monitoring-disabling-grafana.adoc[leveloffset=+1]