diff --git a/modules/monitoring-configuring-a-local-persistent-volume-claim.adoc b/modules/monitoring-configuring-a-persistent-volume-claim.adoc similarity index 60% rename from modules/monitoring-configuring-a-local-persistent-volume-claim.adoc rename to modules/monitoring-configuring-a-persistent-volume-claim.adoc index 0fc74f448d87..8a7bab950c89 100644 --- a/modules/monitoring-configuring-a-local-persistent-volume-claim.adoc +++ b/modules/monitoring-configuring-a-persistent-volume-claim.adoc @@ -3,15 +3,11 @@ // * observability/monitoring/configuring-the-monitoring-stack.adoc :_mod-docs-content-type: PROCEDURE -[id="configuring-a-local-persistent-volume-claim_{context}"] -ifndef::openshift-dedicated,openshift-rosa[] -= Configuring a local persistent volume claim -endif::openshift-dedicated,openshift-rosa[] -ifdef::openshift-dedicated,openshift-rosa[] +[id="configuring-a-persistent-volume-claim_{context}"] + = Configuring a persistent volume claim -endif::openshift-dedicated,openshift-rosa[] -For monitoring components to use a persistent volume (PV), you must configure a persistent volume claim (PVC). +To use a persistent volume (PV) for monitoring components, you must configure a persistent volume claim (PVC). .Prerequisites @@ -52,18 +48,21 @@ metadata: namespace: openshift-monitoring data: config.yaml: | - : + : #<1> volumeClaimTemplate: spec: - storageClassName: + storageClassName: #<2> resources: requests: - storage: + storage: #<3> ---- +<1> Specify the core monitoring component for which you want to configure the PVC. +<2> Specify an existing storage class. If a storage class is not specified, the default storage class is used. +<3> Specify the amount of required storage. + See the link:https://kubernetes.io/docs/concepts/storage/persistent-volumes/#persistentvolumeclaims[Kubernetes documentation on PersistentVolumeClaims] for information on how to specify `volumeClaimTemplate`. + -The following example configures a PVC that claims local persistent storage for the Prometheus instance that monitors core {product-title} components: +The following example configures a PVC that claims persistent storage for the Prometheus instance that monitors core {product-title} components: + [source,yaml] ---- @@ -77,33 +76,11 @@ data: prometheusK8s: volumeClaimTemplate: spec: - storageClassName: local-storage + storageClassName: my-storage-class resources: requests: storage: 40Gi ---- -+ -In the above example, the storage class created by the Local Storage Operator is called `local-storage`. -+ -The following example configures a PVC that claims local persistent storage for Alertmanager: -+ -[source,yaml] ----- -apiVersion: v1 -kind: ConfigMap -metadata: - name: cluster-monitoring-config - namespace: openshift-monitoring -data: - config.yaml: | - alertmanagerMain: - volumeClaimTemplate: - spec: - storageClassName: local-storage - resources: - requests: - storage: 10Gi ----- ** *To configure a PVC for a component that monitors user-defined projects*: endif::openshift-dedicated,openshift-rosa[] @@ -125,58 +102,21 @@ metadata: namespace: openshift-user-workload-monitoring data: config.yaml: | - : + : #<1> volumeClaimTemplate: spec: - storageClassName: + storageClassName: #<2> resources: requests: - storage: + storage: #<3> ---- +<1> Specify the component for user-defined monitoring for which you want to configure the PVC. +<2> Specify an existing storage class. If a storage class is not specified, the default storage class is used. +<3> Specify the amount of required storage. + See the link:https://kubernetes.io/docs/concepts/storage/persistent-volumes/#persistentvolumeclaims[Kubernetes documentation on PersistentVolumeClaims] for information on how to specify `volumeClaimTemplate`. + -The following example configures a PVC that claims -ifndef::openshift-dedicated,openshift-rosa[] -local -endif::openshift-dedicated,openshift-rosa[] -persistent storage for the Prometheus instance that monitors user-defined projects: -+ -[source,yaml] ----- -apiVersion: v1 -kind: ConfigMap -metadata: - name: user-workload-monitoring-config - namespace: openshift-user-workload-monitoring -data: - config.yaml: | - prometheus: - volumeClaimTemplate: - spec: -ifndef::openshift-dedicated,openshift-rosa[] - storageClassName: local-storage -endif::openshift-dedicated,openshift-rosa[] -ifdef::openshift-dedicated,openshift-rosa[] - storageClassName: gp3 -endif::openshift-dedicated,openshift-rosa[] - resources: - requests: - storage: 40Gi ----- -+ -ifndef::openshift-dedicated,openshift-rosa[] -In the above example, the storage class created by the Local Storage Operator is called `local-storage`. -endif::openshift-dedicated,openshift-rosa[] -ifdef::openshift-dedicated,openshift-rosa[] -The above example uses the `gp3` storage class. -endif::openshift-dedicated,openshift-rosa[] -+ -The following example configures a PVC that claims -ifndef::openshift-dedicated,openshift-rosa[] -local -endif::openshift-dedicated,openshift-rosa[] -persistent storage for Thanos Ruler: +The following example configures a PVC that claims persistent storage for Thanos Ruler: + [source,yaml] ---- @@ -190,12 +130,7 @@ data: thanosRuler: volumeClaimTemplate: spec: -ifndef::openshift-dedicated,openshift-rosa[] - storageClassName: local-storage -endif::openshift-dedicated,openshift-rosa[] -ifdef::openshift-dedicated,openshift-rosa[] - storageClassName: gp3 -endif::openshift-dedicated,openshift-rosa[] + storageClassName: my-storage-class resources: requests: storage: 10Gi diff --git a/modules/monitoring-configuring-persistent-storage.adoc b/modules/monitoring-configuring-persistent-storage.adoc index f50b48ec41fa..94f36b5ca4e9 100644 --- a/modules/monitoring-configuring-persistent-storage.adoc +++ b/modules/monitoring-configuring-persistent-storage.adoc @@ -11,9 +11,9 @@ Run cluster monitoring with persistent storage to gain the following benefits: * Protect your metrics and alerting data from data loss by storing them in a persistent volume (PV). As a result, they can survive pods being restarted or recreated. * Avoid getting duplicate notifications and losing silences for alerts when the Alertmanager pods are restarted. -For production environments, it is highly recommended to configure persistent storage. Because of the high IO demands, it is advantageous to use local storage. +For production environments, it is highly recommended to configure persistent storage. -[id="persistent-storage-prerequisites"] +[id="persistent-storage-prerequisites_{context}"] == Persistent storage prerequisites ifdef::openshift-dedicated,openshift-rosa[] @@ -21,21 +21,16 @@ ifdef::openshift-dedicated,openshift-rosa[] endif::openshift-dedicated,openshift-rosa[] ifndef::openshift-dedicated,openshift-rosa[] -* Dedicate sufficient local persistent storage to ensure that the disk does not become full. How much storage you need depends on the number of pods. - -* Verify that you have a persistent volume (PV) ready to be claimed by the persistent volume claim (PVC), one PV for each replica. Because Prometheus and Alertmanager both have two replicas, you need four PVs to support the entire monitoring stack. The PVs are available from the Local Storage Operator, but not if you have enabled dynamically provisioned storage. +* Dedicate sufficient persistent storage to ensure that the disk does not become full. * Use `Filesystem` as the storage type value for the `volumeMode` parameter when you configure the persistent volume. + -[NOTE] -==== -If you use a local volume for persistent storage, do not use a raw block volume, which is described with `volumeMode: Block` in the `LocalVolume` object. Prometheus cannot use raw block volumes. -==== -+ [IMPORTANT] ==== -Prometheus does not support file systems that are not POSIX compliant. +* Do not use a raw block volume, which is described with `volumeMode: Block` in the `PersistentVolume` resource. Prometheus cannot use raw block volumes. + +* Prometheus does not support file systems that are not POSIX compliant. For example, some NFS file system implementations are not POSIX compliant. If you want to use an NFS file system for storage, verify with the vendor that their NFS implementation is fully POSIX compliant. ==== -endif::openshift-dedicated,openshift-rosa[] +endif::openshift-dedicated,openshift-rosa[] \ No newline at end of file diff --git a/modules/monitoring-resizing-a-persistent-storage-volume.adoc b/modules/monitoring-resizing-a-persistent-storage-volume.adoc deleted file mode 100644 index e18c19485862..000000000000 --- a/modules/monitoring-resizing-a-persistent-storage-volume.adoc +++ /dev/null @@ -1,206 +0,0 @@ -// Module included in the following assemblies: -// -// * observability/monitoring/configuring-the-monitoring-stack.adoc - -:_mod-docs-content-type: PROCEDURE -[id="resizing-a-persistent-storage-volume_{context}"] -= Resizing a persistent storage volume - -{product-title} does not support resizing an existing persistent storage volume used by `StatefulSet` resources, even if the underlying `StorageClass` resource used supports persistent volume sizing. -Therefore, even if you update the `storage` field for an existing persistent volume claim (PVC) with a larger size, this setting will not be propagated to the associated persistent volume (PV). - -However, resizing a PV is still possible by using a manual process. If you want to resize a PV for a monitoring component such as Prometheus, Thanos Ruler, or Alertmanager, you can update the appropriate config map in which the component is configured. Then, patch the PVC, and delete and orphan the pods. -Orphaning the pods recreates the `StatefulSet` resource immediately and automatically updates the size of the volumes mounted in the pods with the new PVC settings. -No service disruption occurs during this process. - -.Prerequisites - -* You have installed the OpenShift CLI (`oc`). -* *If you are configuring core {product-title} monitoring components*: -** You have access to the cluster as a user with the `cluster-admin` cluster role. -** You have created the `cluster-monitoring-config` `ConfigMap` object. -** You have configured at least one PVC for core {product-title} monitoring components. -* *If you are configuring components that monitor user-defined projects*: -** You have access to the cluster as a user with the `cluster-admin` cluster role, or as a user with the `user-workload-monitoring-config-edit` role in the `openshift-user-workload-monitoring` project. -** A cluster administrator has enabled monitoring for user-defined projects. -** You have configured at least one PVC for components that monitor user-defined projects. - -.Procedure - -. Edit the `ConfigMap` object: -** *To resize a PVC for a component that monitors core {product-title} projects*: -.. Edit the `cluster-monitoring-config` `ConfigMap` object in the `openshift-monitoring` project: -+ -[source,terminal] ----- -$ oc -n openshift-monitoring edit configmap cluster-monitoring-config ----- - -.. Add a new storage size for the PVC configuration for the component under `data/config.yaml`: -+ -[source,yaml] ----- -apiVersion: v1 -kind: ConfigMap -metadata: - name: cluster-monitoring-config - namespace: openshift-monitoring -data: - config.yaml: | - : <1> - volumeClaimTemplate: - spec: - storageClassName: <2> - resources: - requests: - storage: <3> ----- -<1> Specify the core monitoring component. -<2> Specify the storage class. -<3> Specify the new size for the storage volume. -+ -The following example configures a PVC that sets the local persistent storage to 100 gigabytes for the Prometheus instance that monitors core {product-title} components: -+ -[source,yaml] ----- -apiVersion: v1 -kind: ConfigMap -metadata: - name: cluster-monitoring-config - namespace: openshift-monitoring -data: - config.yaml: | - prometheusK8s: - volumeClaimTemplate: - spec: - storageClassName: local-storage - resources: - requests: - storage: 100Gi ----- -+ -The following example configures a PVC that sets the local persistent storage for Alertmanager to 40 gigabytes: -+ -[source,yaml] ----- -apiVersion: v1 -kind: ConfigMap -metadata: - name: cluster-monitoring-config - namespace: openshift-monitoring -data: - config.yaml: | - alertmanagerMain: - volumeClaimTemplate: - spec: - storageClassName: local-storage - resources: - requests: - storage: 40Gi ----- - -** *To resize a PVC for a component that monitors user-defined projects*: -+ -[NOTE] -==== -You can resize the volumes for the Thanos Ruler and Prometheus instances that monitor user-defined projects. -==== -+ -.. Edit the `user-workload-monitoring-config` `ConfigMap` object in the `openshift-user-workload-monitoring` project: -+ -[source,terminal] ----- -$ oc -n openshift-user-workload-monitoring edit configmap user-workload-monitoring-config ----- - -.. Update the PVC configuration for the monitoring component under `data/config.yaml`: -+ -[source,yaml] ----- -apiVersion: v1 -kind: ConfigMap -metadata: - name: user-workload-monitoring-config - namespace: openshift-user-workload-monitoring -data: - config.yaml: | - : <1> - volumeClaimTemplate: - spec: - storageClassName: <2> - resources: - requests: - storage: <3> ----- -<1> Specify the core monitoring component. -<2> Specify the storage class. -<3> Specify the new size for the storage volume. -+ -The following example configures the PVC size to 100 gigabytes for the Prometheus instance that monitors user-defined projects: -+ -[source,yaml] ----- -apiVersion: v1 -kind: ConfigMap -metadata: - name: user-workload-monitoring-config - namespace: openshift-user-workload-monitoring -data: - config.yaml: | - prometheus: - volumeClaimTemplate: - spec: - storageClassName: local-storage - resources: - requests: - storage: 100Gi ----- -+ -The following example sets the PVC size to 20 gigabytes for Thanos Ruler: -+ -[source,yaml] ----- -apiVersion: v1 -kind: ConfigMap -metadata: - name: user-workload-monitoring-config - namespace: openshift-user-workload-monitoring -data: - config.yaml: | - thanosRuler: - volumeClaimTemplate: - spec: - storageClassName: local-storage - resources: - requests: - storage: 20Gi ----- -+ -[NOTE] -==== -Storage requirements for the `thanosRuler` component depend on the number of rules that are evaluated and how many samples each rule generates. -==== - -. Save the file to apply the changes. The pods affected by the new configuration restart automatically. -+ -[WARNING] -==== -When you save changes to a monitoring config map, the pods and other resources in the related project might be redeployed. The monitoring processes running in that project might also be restarted. -==== - -. Manually patch every PVC with the updated storage request. The following example resizes the storage size for the Prometheus component in the `openshift-monitoring` namespace to 100Gi: -+ -[source,terminal] ----- -$ for p in $(oc -n openshift-monitoring get pvc -l app.kubernetes.io/name=prometheus -o jsonpath='{range .items[*]}{.metadata.name} {end}'); do \ - oc -n openshift-monitoring patch pvc/${p} --patch '{"spec": {"resources": {"requests": {"storage":"100Gi"}}}}'; \ - done - ----- - -. Delete the underlying StatefulSet with the `--cascade=orphan` parameter: -+ -[source,terminal] ----- -$ oc delete statefulset -l app.kubernetes.io/name=prometheus --cascade=orphan ----- diff --git a/modules/monitoring-resizing-a-persistent-volume.adoc b/modules/monitoring-resizing-a-persistent-volume.adoc new file mode 100644 index 000000000000..85798f5c2ea5 --- /dev/null +++ b/modules/monitoring-resizing-a-persistent-volume.adoc @@ -0,0 +1,145 @@ +// Module included in the following assemblies: +// +// * observability/monitoring/configuring-the-monitoring-stack.adoc + +:_mod-docs-content-type: PROCEDURE +[id="resizing-a-persistent-volume_{context}"] += Resizing a persistent volume + +You can resize a persistent volume (PV) for monitoring components, such as Prometheus, Thanos Ruler, or Alertmanager. You need to manually expand a persistent volume claim (PVC), and then update the config map in which the component is configured. + +[IMPORTANT] +==== +You can only expand the size of the PVC. Shrinking the storage size is not possible. +==== + +.Prerequisites + +* You have installed the OpenShift CLI (`oc`). +* *If you are configuring core {product-title} monitoring components*: +** You have access to the cluster as a user with the `cluster-admin` cluster role. +** You have created the `cluster-monitoring-config` `ConfigMap` object. +** You have configured at least one PVC for core {product-title} monitoring components. +* *If you are configuring components that monitor user-defined projects*: +** You have access to the cluster as a user with the `cluster-admin` cluster role, or as a user with the `user-workload-monitoring-config-edit` role in the `openshift-user-workload-monitoring` project. +** A cluster administrator has enabled monitoring for user-defined projects. +** You have configured at least one PVC for components that monitor user-defined projects. + +.Procedure + +. Manually expand a PVC with the updated storage request. For more information, see "Expanding persistent volume claims (PVCs) with a file system" in _Expanding persistent volumes_. + +. Edit the `ConfigMap` object: +** *If you are configuring core {product-title} monitoring components*: +.. Edit the `cluster-monitoring-config` `ConfigMap` object in the `openshift-monitoring` project: ++ +[source,terminal] +---- +$ oc -n openshift-monitoring edit configmap cluster-monitoring-config +---- + +.. Add a new storage size for the PVC configuration for the component under `data/config.yaml`: ++ +[source,yaml] +---- +apiVersion: v1 +kind: ConfigMap +metadata: + name: cluster-monitoring-config + namespace: openshift-monitoring +data: + config.yaml: | + : #<1> + volumeClaimTemplate: + spec: + resources: + requests: + storage: #<2> +---- +<1> The component for which you want to change the storage size. +<2> Specify the new size for the storage volume. It must be greater than the previous value. ++ +The following example sets the new PVC request to 100 gigabytes for the Prometheus instance that monitors core {product-title} components: ++ +[source,yaml] +---- +apiVersion: v1 +kind: ConfigMap +metadata: + name: cluster-monitoring-config + namespace: openshift-monitoring +data: + config.yaml: | + prometheusK8s: + volumeClaimTemplate: + spec: + resources: + requests: + storage: 100Gi +---- + +** *If you are configuring components that monitor user-defined projects*: ++ +[NOTE] +==== +You can resize the volumes for the Thanos Ruler and for instances of Alertmanager and Prometheus that monitor user-defined projects. +==== ++ +.. Edit the `user-workload-monitoring-config` `ConfigMap` object in the `openshift-user-workload-monitoring` project: ++ +[source,terminal] +---- +$ oc -n openshift-user-workload-monitoring edit configmap user-workload-monitoring-config +---- + +.. Update the PVC configuration for the monitoring component under `data/config.yaml`: ++ +[source,yaml] +---- +apiVersion: v1 +kind: ConfigMap +metadata: + name: user-workload-monitoring-config + namespace: openshift-user-workload-monitoring +data: + config.yaml: | + : #<1> + volumeClaimTemplate: + spec: + resources: + requests: + storage: #<2> +---- +<1> The component for which you want to change the storage size. +<2> Specify the new size for the storage volume. It must be greater than the previous value. ++ +The following example sets the new PVC request to 20 gigabytes for Thanos Ruler: ++ +[source,yaml] +---- +apiVersion: v1 +kind: ConfigMap +metadata: + name: user-workload-monitoring-config + namespace: openshift-user-workload-monitoring +data: + config.yaml: | + thanosRuler: + volumeClaimTemplate: + spec: + resources: + requests: + storage: 20Gi +---- ++ +[NOTE] +==== +Storage requirements for the `thanosRuler` component depend on the number of rules that are evaluated and how many samples each rule generates. +==== + +. Save the file to apply the changes. ++ +[WARNING] +==== +When you save changes to a monitoring config map, the pods and other resources in the related project are redeployed. The monitoring processes running in that project are restarted. +==== diff --git a/modules/monitoring-specifying-limits-and-requests-for-monitoring-components.adoc b/modules/monitoring-specifying-limits-and-requests-for-monitoring-components.adoc index 87c1f7ca2503..ddfa6b48031b 100644 --- a/modules/monitoring-specifying-limits-and-requests-for-monitoring-components.adoc +++ b/modules/monitoring-specifying-limits-and-requests-for-monitoring-components.adoc @@ -146,3 +146,6 @@ When you save changes to the `cluster-monitoring-config` config map, the pods an The running monitoring processes in that project might also restart. ==== +[role="_additional-resources"] +.Additional resources +* link:https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/#requests-and-limits[Kubernetes requests and limits documentation] diff --git a/observability/monitoring/configuring-the-monitoring-stack.adoc b/observability/monitoring/configuring-the-monitoring-stack.adoc index a19216ac9296..57dc2e0f9977 100644 --- a/observability/monitoring/configuring-the-monitoring-stack.adoc +++ b/observability/monitoring/configuring-the-monitoring-stack.adoc @@ -157,20 +157,23 @@ You can configure these limits and requests for core platform monitoring compone include::modules/monitoring-about-specifying-limits-and-requests-for-monitoring-components.adoc[leveloffset=+2] include::modules/monitoring-specifying-limits-and-requests-for-monitoring-components.adoc[leveloffset=+2] -[role="_additional-resources"] -.Additional resources -* link:https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/#requests-and-limits[Kubernetes requests and limits documentation] - // Enabling a dedicated service monitor include::modules/monitoring-configuring-dedicated-service-monitors.adoc[leveloffset=+1] include::modules/monitoring-enabling-a-dedicated-service-monitor.adoc[leveloffset=+2] // Configuring persistent storage include::modules/monitoring-configuring-persistent-storage.adoc[leveloffset=+1] -include::modules/monitoring-configuring-a-local-persistent-volume-claim.adoc[leveloffset=+2] +include::modules/monitoring-configuring-a-persistent-volume-claim.adoc[leveloffset=+2] + ifndef::openshift-dedicated,openshift-rosa[] -include::modules/monitoring-resizing-a-persistent-storage-volume.adoc[leveloffset=+2] +include::modules/monitoring-resizing-a-persistent-volume.adoc[leveloffset=+2] + +[role="_additional-resources"] +.Additional resources +* xref:../../scalability_and_performance/recommended-performance-scale-practices/recommended-infrastructure-practices.adoc#prometheus-database-storage-requirements_recommended-infrastructure-practices[Prometheus database storage requirements] +* xref:../../storage/expanding-persistent-volumes.adoc#expanding-pvc-filesystem_expanding-persistent-volumes[Expanding persistent volume claims (PVCs) with a file system] endif::openshift-dedicated,openshift-rosa[] + include::modules/monitoring-modifying-retention-time-and-size-for-prometheus-metrics-data.adoc[leveloffset=+2] include::modules/monitoring-modifying-the-retention-time-for-thanos-ruler-metrics-data.adoc[leveloffset=+2] @@ -182,7 +185,6 @@ ifndef::openshift-dedicated,openshift-rosa[] * xref:../../scalability_and_performance/optimization/optimizing-storage.adoc#optimizing-storage[Recommended configurable storage technology] * xref:../../storage/understanding-persistent-storage.adoc#understanding-persistent-storage[Understanding persistent storage] * xref:../../scalability_and_performance/optimization/optimizing-storage.adoc#optimizing-storage[Optimizing storage] -* xref:../../storage/persistent_storage/persistent_storage_local/persistent-storage-local.adoc#persistent-storage-using-local-volume[Configure local persistent storage] * xref:../../observability/monitoring/enabling-monitoring-for-user-defined-projects.adoc#enabling-monitoring-for-user-defined-projects[Enabling monitoring for user-defined projects] endif::openshift-dedicated,openshift-rosa[] ifdef::openshift-dedicated,openshift-rosa[]