Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[enterprise-4.9] MON-1687 - Add new monitoring content for remote write configuration #37075

Merged
merged 1 commit into from
Oct 5, 2021
Merged

[enterprise-4.9] MON-1687 - Add new monitoring content for remote write configuration #37075

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
215 changes: 215 additions & 0 deletions modules/monitoring-configuring-remote-write.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,215 @@
// Module included in the following assemblies:
//
// * monitoring/configuring-the-monitoring-stack.adoc

[id="configuring_remote_write_storage_{context}"]
= Configuring remote write storage

You can configure remote write storage to enable Prometheus to send ingested metrics to remote systems for long-term storage.
Doing so has no impact on how or for how long Prometheus stores metrics.

.Prerequisites

* *If you are configuring core {product-title} monitoring components:*
** You have access to the cluster as a user with the `cluster-admin` role.
** You have created the `cluster-monitoring-config` `ConfigMap` object.
* *If you are configuring components that monitor user-defined projects:*
** You have access to the cluster as a user with the `cluster-admin` role or as a user with the `user-workload-monitoring-config-edit` role in the `openshift-user-workload-monitoring` project.
** You have created the `user-workload-monitoring-config` `ConfigMap` object.
* You have installed the OpenShift CLI (`oc`).
* You have set up a remote write compatible endpoint (such as Thanos) and know the endpoint URL.
See the link:https://prometheus.io/docs/operating/integrations/#remote-endpoints-and-storage[Prometheus remote endpoints and storage documentation] for information about endpoints that are compatible with the remote write feature.
* You have set up authentication credentials for the remote write endpoint.
+
[CAUTION]
====
To reduce security risks, avoid sending metrics to an endpoint via unencrypted HTTP or without using authentication.
====

.Procedure

. Edit the `cluster-monitoring-config` `ConfigMap` object in the `openshift-monitoring` project:
+
[source,terminal]
----
$ oc -n openshift-monitoring edit configmap cluster-monitoring-config
----

. Add a `remoteWrite:` section under `data/config.yaml/prometheusK8s`.

. Add an endpoint URL and authentication credentials in this section:
+
[source,yaml]
----
apiVersion: v1
kind: ConfigMap
metadata:
name: cluster-monitoring-config
namespace: openshift-monitoring
data:
config.yaml: |
prometheusK8s:
remoteWrite:
- url: "https://remote-write.endpoint"
<endpoint_authentication_credentials>
----
+
For `endpoint_authentication_credentials` substitute the credentials for the endpoint.
Currently supported authentication methods are basic authentication (`basicAuth`) and client TLS (`tlsConfig`) authentication.
+
* The following example configures basic authentication:
+
[source,yaml]
----
basicAuth:
username:
<usernameSecret>
password:
<passwordSecret>
----
Substitute `<usernameSecret>` and `<passwordSecret>` accordingly.
+
The following sample shows basic authentication configured with `remoteWriteAuth` for the `name` values and `user` and `password` for the `key` values. These values contain the endpoint authentication credentials:
+
[source,yaml]
----
apiVersion: v1
kind: ConfigMap
metadata:
name: cluster-monitoring-config
namespace: openshift-monitoring
data:
config.yaml: |
prometheusK8s:
remoteWrite:
- url: "https://remote-write.endpoint"
basicAuth:
username:
name: remoteWriteAuth
key: user
password:
name: remoteWriteAuth
key: password
----
+
* The following example configures client TLS authentication:
+
[source,yaml]
----
tlsConfig:
ca:
<caSecret>
cert:
<certSecret>
keySecret:
<keySecret>
----
Substitute `<caSecret>`, `<certSecret>`, and `<keySecret>` accordingly.
+
The following sample shows a TLS authentication configuration using `selfsigned-mtls-bundle` for the `name` values and `ca.crt` for the `ca` `key` value, `client.crt` for the `cert` `key` value, and `client.key` for the `keySecret` `key` value:
+
[source,yaml]
----
apiVersion: v1
kind: ConfigMap
metadata:
name: cluster-monitoring-config
namespace: openshift-monitoring
data:
config.yaml: |
prometheusK8s:
remoteWrite:
- url: "https://remote-write.endpoint"
tlsConfig:
ca:
secret:
name: selfsigned-mtls-bundle
key: ca.crt
cert:
secret:
name: selfsigned-mtls-bundle
key: client.crt
keySecret:
name: selfsigned-mtls-bundle
key: client.key
----

. Add write relabel configuration values after the authentication credentials:
+
[source,yaml]
----
apiVersion: v1
kind: ConfigMap
metadata:
name: cluster-monitoring-config
namespace: openshift-monitoring
data:
config.yaml: |
prometheusK8s:
remoteWrite:
- url: "https://remote-write.endpoint"
<endpoint_authentication_credentials>
<write_relabel_configs>
----
+
For `<write_relabel_configs>` substitute a list of write relabel configurations for metrics that you want to send to the remote endpoint.
+
The following sample shows how to forward a single metric called `my_metric`:
+
[source,yaml]
----
apiVersion: v1
kind: ConfigMap
metadata:
name: cluster-monitoring-config
namespace: openshift-monitoring
data:
config.yaml: |
prometheusK8s:
remoteWrite:
- url: "https://remote-write.endpoint"
writeRelabelConfigs:
- source_labels: [__name__]
regex: 'my_metric'
action: keep

----
+
See the link:https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config[Prometheus relabel_config documentation] for information about write relabel configuration options.

. If required, configure remote write for the Prometheus instance that monitors user-defined projects by changing the `name` and `namespace` `metadata` values as follows:
+
[source,yaml]
----
apiVersion: v1
kind: ConfigMap
metadata:
name: user-workload-monitoring-config
namespace: openshift-user-workload-monitoring
data:
config.yaml: |
prometheus:
remoteWrite:
- url: "https://remote-write.endpoint"
<endpoint_authentication_credentials>
<write_relabel_configs>
----
+
[NOTE]
====
The Prometheus config map component is called `prometheusK8s` in the `cluster-monitoring-config` `ConfigMap` object and `prometheus` in the `user-workload-monitoring-config` `ConfigMap` object.
====

. Save the file to apply the changes to the `ConfigMap` object.
The pods affected by the new configuration restart automatically.
+
[NOTE]
====
Configurations applied to the `user-workload-monitoring-config` `ConfigMap` object are not activated unless a cluster administrator has enabled monitoring for user-defined projects.
====
+
[WARNING]
====
Saving changes to a monitoring `ConfigMap` object might redeploy the pods and other resources in the related project. Saving changes might also restart the running monitoring processes in that project.
====

9 changes: 9 additions & 0 deletions monitoring/configuring-the-monitoring-stack.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -105,6 +105,15 @@ include::modules/monitoring-modifying-retention-time-for-prometheus-metrics-data
* xref:../storage/understanding-persistent-storage.adoc#understanding-persistent-storage[Understanding persistent storage]
* xref:../scalability_and_performance/optimizing-storage.adoc#optimizing-storage[Optimizing storage]

// Configuring remote write storage for Prometheus
include::modules/monitoring-configuring-remote-write.adoc[leveloffset=+1]

.Additional resources

* See link:https://prometheus.io/docs/operating/integrations/#remote-endpoints-and-storage[Setting up remote write compatible endpoints] for steps to create a remote write compatible endpoint (such as Thanos).
* See link:https://prometheus.io/docs/practices/remote_write/#remote-write-tuning[Tuning remote write settings] for information about how to optimize remote write settings for different use cases.
* For information about additional optional fields, please refer to the API documentation.

// Managing scrape sample limits for user-defined projects
include::modules/monitoring-limiting-scrape-samples-in-user-defined-projects.adoc[leveloffset=+1]
include::modules/monitoring-setting-a-scrape-sample-limit-for-user-defined-projects.adoc[leveloffset=+2]
Expand Down