Skip to content

[Metrics] Document recent opentelemetry improvements #104

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
KennanHunter wants to merge 4 commits into
base: main
Choose a base branch
from
Open

[Metrics] Document recent opentelemetry improvements #104

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
6164ce7
document OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE behavior
KennanHunter Jul 31, 2026
dd4d69b
improve documentation for temporality as a toml value
KennanHunter Aug 2, 2026
2f59fab
describe locking metrics
KennanHunter Aug 2, 2026
8119380
include DataDog and Grafana example links in metrics
KennanHunter Aug 2, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Empty file modified dev.sh
100644 → 100755
View file
Open in desktop
Empty file.
23 changes: 23 additions & 0 deletions docs/configuration/pgdog.toml/otel.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -67,6 +67,27 @@ How often, in milliseconds, to push metrics to the OTLP endpoint.

_Default:_ `10000`

### `temporality_preference`

How exported metric points describe their value over time. See the [OpenTelemetry metric points spec](https://opentelemetry.io/docs/specs/otel/metrics/data-model/#metric-points).

| Value | Behavior |
| ------------ | --------------------------------------------------------------- |
| `Cumulative` | Points report the value accumulated since the exporter started. |
| `Delta` | Points report the change since the last export. |

<!-- This value is supported but currently meaningless. -->
<!--| `LowMemory` | Delta for sums, cumulative for histograms; minimizes exporter memory. |-->

_Default:_ `Cumulative`, or `Delta` when `datadog_api_key` is set.

Values are case-insensitive.

!!! warning "Datadog"
Sending `Cumulative` sums and histograms to Datadog is stateful and lossy: all points on a timeseries must reach the same Agent/exporter (constraining how you scale collectors), the first point of a new series may be dropped (causing gaps on restart), and histogram min/max may be missing or approximated. See [Datadog's OTLP delta temporality guide](https://docs.datadoghq.com/opentelemetry/guide/otlp_delta_temporality/?tab=python#implications-of-using-cumulative-aggregation-temporality).

PgDog logs a warning when `datadog_api_key` is set alongside `Cumulative`. Set `IGNORE_DATADOG_CUMULATIVE_WARNING=1` to silence it.

## Environment variables

PgDog honors the standard OpenTelemetry environment variables. When set, they override the
Expand All @@ -79,9 +100,11 @@ corresponding `pgdog.toml` values (or supply a value when the setting is omitted
| `OTEL_METRIC_EXPORT_INTERVAL` | Push interval in milliseconds. Equivalent to `push_interval`. |
| `OTEL_SERVICE_NAME` | Sets the `service.name` resource attribute. Takes precedence over any value set via `OTEL_RESOURCE_ATTRIBUTES`. _Default:_ `pgdog`. |
| `OTEL_RESOURCE_ATTRIBUTES` | Comma-separated `key=value` pairs added as resource attributes on every metric (e.g. `env=prod,region=us-east-1`). Values may be percent-encoded. |
| `OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE` | Aggregation temporality for exported metrics. Equivalent to `temporality_preference`. |
| `PGDOG_OTEL_NAMESPACE` | Metric name prefix. Equivalent to `namespace`. |
| `DD_API_KEY` | Datadog API key. Equivalent to `datadog_api_key`. |


PgDog automatically sets the following resource attributes on every exported metric:
`service.name` (defaults to `pgdog`), `service.instance.id`, and `host.name`. Use
`OTEL_RESOURCE_ATTRIBUTES` to add or override any of these.
7 changes: 5 additions & 2 deletions docs/features/metrics.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,8 +34,9 @@ You can connect to the admin database using any PostgreSQL client. It supports c

## OpenMetrics

[OpenMetrics](https://openmetrics.io/) is a standard for displaying metrics that can be ingested by a multitude of agents, e.g., Datadog, Prometheus, etc. The metrics are shown in a human-readable text format
and available from an HTTP endpoint.
[OpenMetrics](https://openmetrics.io/) is a standard for displaying metrics that can be ingested by a multitude of agents, e.g., Datadog, Prometheus, etc. The metrics are shown in a human-readable text format and available from an HTTP endpoint.

We maintain multiple examples of OpenMetrics integrations, including [DataDog](https://github.com/pgdogdev/pgdog/tree/main/examples/datadog) and [Grafana using Prometheus](https://github.com/pgdog/pgdog/tree/main/examples/grafana_prometheus).

### Configuration

Expand Down Expand Up @@ -146,9 +147,11 @@ The following metrics are exported via the OpenMetrics endpoint:
|------------|-------------|------|
| `clients` | Total number of connected clients. | `gauge` |
| `cl_waiting` | Clients waiting for a connection from a pool. | `gauge` |
| `cl_locked` | Clients currently locked to a specific connection. | `gauge` |
| `sv_active` | Servers currently serving client requests. | `gauge` |
| `sv_idle` | Servers available for clients to use. | `gauge` |
| `sv_idle_xact` | Servers currently idle in transaction. | `gauge` |
| `sv_locked` | Servers currently locked. | `gauge` |
| `maxwait` | How long the first (oldest) client in the queue has waited (in seconds). | `gauge` |
| `errors` | Errors that connections in the pool have experienced. | `counter` |
| `out_of_sync` | Connections that have been returned to the pool in a broken state. | `counter` |
Expand Down