Skip to content

[enterprise-4.15] TRACING-4562: Improve Prometheus port and services documentation #82912

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Oct 2, 2024
Merged

[enterprise-4.15] TRACING-4562: Improve Prometheus port and services documentation #82912

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 11 additions & 8 deletions modules/otel-config-send-metrics-monitoring-stack.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,12 +6,7 @@
[id="configuration-for-sending-metrics-to-the-monitoring-stack_{context}"]
= Configuration for sending metrics to the monitoring stack

One of two following custom resources (CR) configures the sending of metrics to the monitoring stack:

* OpenTelemetry Collector CR
* Prometheus `PodMonitor` CR

A configured OpenTelemetry Collector CR can create a Prometheus `ServiceMonitor` CR for scraping the Collector's pipeline metrics and the enabled Prometheus exporters.
You can configure the `OpenTelemetryCollector` custom resource (CR) to create a Prometheus `ServiceMonitor` CR or a `PodMonitor` CR for a sidecar deployment. A `ServiceMonitor` can scrape Collector's internal metrics endpoint and Prometheus exporter metrics endpoints.

.Example of the OpenTelemetry Collector CR with the Prometheus exporter
[source,yaml]
Expand All @@ -38,7 +33,15 @@ spec:
receivers: [otlp]
exporters: [prometheus]
----
<1> Configures the Operator to create the Prometheus `ServiceMonitor` CR to scrape the Collector's internal metrics endpoint and Prometheus exporter metric endpoints. The metrics will be stored in the OpenShift monitoring stack.
<1> Configures the {OTELOperator} to create the Prometheus `ServiceMonitor` CR or `PodMonitor` CR to scrape the Collector's internal metrics endpoint and the Prometheus exporter metrics endpoints.

[NOTE]
====
Setting `enableMetrics` to `true` creates the following two `ServiceMonitor` instances:

* One `ServiceMonitor` instance for the `<instance_name>-collector-monitoring` service. This `ServiceMonitor` instance scrapes the Collector's internal metrics.
* One `ServiceMonitor` instance for the `<instance_name>-collector` service. This `ServiceMonitor` instance scrapes the metrics exposed by the Prometheus exporter instances.
====

Alternatively, a manually created Prometheus `PodMonitor` CR can provide fine control, for example removing duplicated labels added during Prometheus scraping.

Expand Down Expand Up @@ -71,4 +74,4 @@ spec:
----
<1> The name of the OpenTelemetry Collector CR.
<2> The name of the internal metrics port for the OpenTelemetry Collector. This port name is always `metrics`.
<3> The name of the Prometheus exporter port for the OpenTelemetry Collector.
<3> The name of the Prometheus exporter port for the OpenTelemetry Collector.
45 changes: 23 additions & 22 deletions observability/otel/otel-collector/otel-collector-exporters.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -153,41 +153,42 @@ include::snippets/technology-preview.adoc[]
[source,yaml]
----
# ...
ports:
- name: promexporter # <1>
port: 8889
protocol: TCP
config: |
exporters:
prometheus:
endpoint: 0.0.0.0:8889 # <2>
tls: # <3>
endpoint: 0.0.0.0:8889 # <1>
tls: # <2>
ca_file: ca.pem
cert_file: cert.pem
key_file: key.pem
namespace: prefix # <4>
const_labels: # <5>
namespace: prefix # <3>
const_labels: # <4>
label1: value1
enable_open_metrics: true # <6>
resource_to_telemetry_conversion: # <7>
enable_open_metrics: true # <5>
resource_to_telemetry_conversion: # <6>
enabled: true
metric_expiration: 180m # <8>
add_metric_suffixes: false # <9>
metric_expiration: 180m # <7>
add_metric_suffixes: false # <8>
service:
pipelines:
metrics:
exporters: [prometheus]
# ...
----
<1> Exposes the Prometheus port from the Collector pod and service. You can enable scraping of metrics by Prometheus by using the port name in `ServiceMonitor` or `PodMonitor` custom resource.
<2> The network endpoint where the metrics are exposed.
<3> The server-side TLS configuration. Defines paths to TLS certificates.
<4> If set, exports metrics under the provided value. No default.
<5> Key-value pair labels that are applied for every exported metric. No default.
<6> If `true`, metrics are exported using the OpenMetrics format. Exemplars are only exported in the OpenMetrics format and only for histogram and monotonic sum metrics such as `counter`. Disabled by default.
<7> If `enabled` is `true`, all the resource attributes are converted to metric labels by default. Disabled by default.
<8> Defines how long metrics are exposed without updates. The default is `5m`.
<9> Adds the metrics types and units suffixes. Must be disabled if the monitor tab in Jaeger console is enabled. The default is `true`.
<1> The network endpoint where the metrics are exposed. The {OTELOperator} automatically exposes the port specified in the `endpoint` field to the `<instance_name>-collector` service.
<2> The server-side TLS configuration. Defines paths to TLS certificates.
<3> If set, exports metrics under the provided value.
<4> Key-value pair labels that are applied for every exported metric.
<5> If `true`, metrics are exported by using the OpenMetrics format. Exemplars are only exported in the OpenMetrics format and only for histogram and monotonic sum metrics such as `counter`. Disabled by default.
<6> If `enabled` is `true`, all the resource attributes are converted to metric labels. Disabled by default.
<7> Defines how long metrics are exposed without updates. The default is `5m`.
<8> Adds the metrics types and units suffixes. Must be disabled if the monitor tab in the Jaeger console is enabled. The default is `true`.


[NOTE]
====
When the `spec.observability.metrics.enableMetrics` field in the `OpenTelemetryCollector` custom resource (CR) is set to `true`, the `OpenTelemetryCollector` CR automatically creates a Prometheus `ServiceMonitor` or `PodMonitor` CR to enable Prometheus to scrape your metrics.
====

[id="prometheus-remote-write-exporter_{context}"]
== Prometheus Remote Write Exporter
Expand Down Expand Up @@ -276,4 +277,4 @@ include::snippets/technology-preview.adoc[]
[role="_additional-resources"]
[id="additional-resources_otel-collector-exporters_{context}"]
== Additional resources
* link:https://opentelemetry.io/docs/specs/otlp/[OpenTelemetry Protocol (OTLP) documentation]
* link:https://opentelemetry.io/docs/specs/otlp/[OpenTelemetry Protocol (OTLP) documentation]
25 changes: 19 additions & 6 deletions observability/otel/otel-configuring-otelcol-metrics.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,11 +4,22 @@
include::_attributes/common-attributes.adoc[]
:context: otel-configuring-metrics

//[id="setting-up-monitoring-for-otel"]
//== Setting up monitoring for the {OTELShortName}
//The {OTELOperator} supports monitoring and alerting of each OpenTelemtry Collector instance and exposes upgrade and operational metrics about the Operator itself.
The following list shows some of these metrics:

You can enable metrics and alerts of OpenTelemetry Collector instances.
* Collector memory usage
* CPU utilization
* Number of active traces and spans processed
* Dropped spans, logs, or metrics
* Exporter and receiver statistics

The {OTELOperator} automatically creates a service named `<instance_name>-collector-monitoring` that exposes the Collector's internal metrics. This service listens on port `8888` by default.

You can use these metrics for monitoring the Collector's performance, resource consumption, and other internal behaviors. You can also use a Prometheus instance or another monitoring tool to scrape these metrics from the mentioned `<instance_name>-collector-monitoring` service.

[NOTE]
====
When the `spec.observability.metrics.enableMetrics` field in the `OpenTelemetryCollector` custom resource (CR) is set to `true`, the `OpenTelemetryCollector` CR automatically creates a Prometheus `ServiceMonitor` or `PodMonitor` CR to enable Prometheus to scrape your metrics.
====

.Prerequisites

Expand All @@ -34,7 +45,9 @@ spec:

You can use the *Administrator* view of the web console to verify successful configuration:

* Go to *Observe* -> *Targets*, filter by *Source: User*, and check that the *ServiceMonitors* in the `opentelemetry-collector-<instance_name>` format have the *Up* status.
. Go to *Observe* -> *Targets*.
. Filter by *Source: User*.
. Check that the *ServiceMonitors* or *PodMonitors* in the `opentelemetry-collector-<instance_name>` format have the *Up* status.

.Additional resources
* xref:../../observability/monitoring/enabling-monitoring-for-user-defined-projects.adoc#enabling-monitoring-for-user-defined-projects[Enabling monitoring for user-defined projects]
* xref:../../observability/monitoring/enabling-monitoring-for-user-defined-projects.adoc#enabling-monitoring-for-user-defined-projects[Enabling monitoring for user-defined projects]