[Metrics] Piping OTEL metrics directly to Prometheus causes two errors

[KennanHunter]

PgDog version
c4f63fd6 latest main

Description
I was testing some OTEL code locally and pointed pgdog straight at Prometheus's native OTLP receiver (--web.enable-otlp-receiver). Hit two errors.

Pgdog's OTEL exporter looks like it's built for Datadog, and Prometheus users are supposed to hit the OpenMetrics /metrics endpoint instead. Both Datadog and the OpenTelemetry Collector tolerate these errors, so this only surfaces if you point OTLP at Prometheus directly. I'm filing anyway because the failure was confusing to me, the fixes are pretty simple, and selfishly I'm a big Prometheus fan.

1. DELTA temporality on counters is rejected by Prometheus

pgdog hardcodes aggregation_temporality: 1 (DELTA) for counters at pgdog/src/stats/otel.rs:285. Prometheus's OTLP receiver is CUMULATIVE-only by default and rejects the whole batch:

invalid temporality and type combination for metric "pgdog.total_server_errors"

Because Prometheus 400s the entire request, every gauge sharing that batch with a DELTA counter gets dropped with it.

There's a workaround on Prometheus's side: --enable-feature=otlp-deltatocumulative, but it's nice if users don't have to. We already calculate the delta using an accumulated value, so if we just return the accumulated value it should be a pretty fast fix.

The best fix is to make us match the exporter spec and switch between delta/cumulative based on environment variable as described in the spec

...MUST set temporality preference to Cumulative for all instrument kinds by default.

The exporter MUST configure the default aggregation temporality on the basis of instrument kind using the OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE variable as described below.

2. target_info out-of-order writes from parallel batching

Out of order sample from remote write ... series="{__name__=\"target_info\", host_name=\"...\", instance=\"...\", job=\"pgdog\"}"

The push loop in pgdog/src/stats/otel_exporter.rs splits metrics into batches of 10 and fires them concurrently with futures::future::join_all. Each batch calls otel::build_request(), which grabs its own now_nanos().

Every batch carries the same OTLP Resource block, so Prometheus's OTLP receiver synthesizes one target_info write per batch. If the same target_info is sent to prometheus with a timestamp less than a previous one, Prometheus rejects them as out-of-order for the target_info series. I believe actual metric samples aren't affected since each one only appears in one batch per cycle. Only real downside is it fills up the logs with warnings.

This one is a fast fix, just collect now_nanos() before you batch and reuse the same timestamp for the entire batch.

Reproduction

I setup a local docker-compose.yml to query otel metrics. Prometheus is pointed at pgdog's OTLP endpoint with delta-to-cumulative enabled so the first error class doesn't mask the second.

# docker-compose.yml
services:
  pgdog:
    build: .
    environment:
      RUST_LOG: debug
      OTEL_EXPORTER_OTLP_ENDPOINT: http://prometheus:9090/api/v1/otlp/v1/metrics
      OTEL_METRIC_EXPORT_INTERVAL: "2000"
    depends_on: [prometheus]
  prometheus:
    image: prom/prometheus:latest
    command:
      - --config.file=/etc/prometheus/prometheus.yml
      - --web.enable-otlp-receiver
      - --enable-feature=otlp-deltatocumulative
      - --log.level=debug
    ports: [9090:9090]

docker compose up --build -d
sleep 15
docker compose logs prometheus | grep -E "temporality|Out of order"

Without --enable-feature=otlp-deltatocumulative you see error #1 (invalid temporality). With it enabled you see error #2 (target_info out of order). Every push cycle produces both classes until the flag is set; only #2 remains after.

Logs

prometheus-1  | time=2026-07-30T00:05:57.608Z level=ERROR source=write_handler.go:323 msg="Out of order sample from remote write" component=web err="out of order sample" series="{__name__=\"target_info\", host_name=\"047695bd73aa\", instance=\"90fcc956\", job=\"pgdog\"}" timestamp=1785369957605
pgdog-1       | 2026-07-30T00:05:57.608661Z TRACE shouldn't retry!
prometheus-1  | time=2026-07-30T00:05:57.608Z level=WARN source=write_handler.go:619 msg="Error translating OTLP metrics to Prometheus write request" component=web err="invalid temporality and type combination for metric \"pgdog.two_pc_recovered_total\""
pgdog-1       | 2026-07-30T00:05:57.608678Z TRACE put; add idle connection for ("http", prometheus:9090)

prometheus-1  | time=2026-07-30T00:05:57.608Z level=ERROR source=write_handler.go:323 msg="Out of order sample from remote write" component=web err="out of order sample" series="{__name__=\"target_info\", host_name=\"047695bd73aa\", instance=\"90fcc956\", job=\"pgdog\"}" timestamp=1785369957605
pgdog-1       | 2026-07-30T00:05:57.608688Z DEBUG pooling idle connection for ("http", prometheus:9090)
pgdog-1       | 2026-07-30T00:05:57.608694Z TRACE shouldn't retry!

pgdog-1       | 2026-07-30T00:05:57.608712Z TRACE put; add idle connection for ("http", prometheus:9090)
pgdog-1       | 2026-07-30T00:05:57.608718Z DEBUG pooling idle connection for ("http", prometheus:9090)
pgdog-1       | 2026-07-30T00:05:57.608723Z TRACE shouldn't retry!
pgdog-1       | 2026-07-30T00:05:57.608758Z  WARN otel exporter: endpoint returned 400 Bad Request: out of order sample

Configuration
Used ./docker/pgdog.toml and ./docker/users.toml, with pgdog built from source (build: . in docker-compose.yml) so OTEL_EXPORTER_OTLP_ENDPOINT was picked up from the environment.

The env var only takes effect when an [otel] section is also present in pgdog.toml. With no [otel] block, serde uses Otel::default() and never consults the env-var fallbacks in pgdog-config/src/otel.rs:71-101. It's a little unintuitive and took me a few minutes to figure out, maybe worth changing. Maybe we warn if the env var is present but [otel] isn't.

Happy to fix either or both errors if you guys think it's worth it, don't want to clog review if out of scope.