[Metrics] Datapoints follow OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE

[KennanHunter]

Closes #1289

Wires OTEL metric exports up to OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE so operators can pick delta, cumulative, or low-memory temporality per the spec.

Opinions

• Case-insensitive parsing of OtelTemporalityPreference from both the env var and the [otel] config section, aligned with OTEL's spec (env var read, custom Deserialize).
• Migrated OpenMetric to a real enum for metric type instead of magic strings, so adding future variants like histogram stays ergonomic (OpenMetricType).
• Refactored the OTLP data-point calculation to make the relationship between metric type, temporality, and the computed value explicit (value_for_data_point, wrap_data_points, build_attributes).

I think before merge some level of integration testing could be nice. How would you go about that @levish0 ? We could also add Prometheus+graphana in the examples folder and include a graphana dashboard script.

The default export has been changed from Delta to Cumulative to match spec. I assume DataDog should have no issue with that, but it might be good for someone familiar with it to test it.

[codecov]

Codecov Report

❌ Patch coverage is 98.07692% with 9 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
pgdog-config/src/otel.rs	95.86%	5 Missing ⚠️
pgdog/src/stats/otel.rs	98.31%	4 Missing ⚠️

📢 Thoughts on this report? Let us know!

[KennanHunter]

Also added support to send the starting timestamp for a metric, in accordance with OTEL spec.

[KennanHunter]

Looking at https://docs.datadoghq.com/opentelemetry/guide/otlp_delta_temporality/?tab=python#overview

Datadog works best with delta aggregation temporality for monotonic sums, histograms, and exponential histograms.

I don't know quite what "works best" means, but it might be worth going out of spec and using delta as the default to just not break stuff. Weird.

[KennanHunter]

Matching docs update

pgdogdev/docs@main...KennanHunter:docs-1:otel-temporality-docs

[KennanHunter]

Prometheus working well locally

[levkk]

Nice! I will review tomorrow first thing!

I assume DataDog should have no issue with that, but it might be good for someone familiar with it to test it.

IIRC, last time I tested this with DD, it only worked with delta. So, maybe we can default it it to delta if datadog_api_key is set in the config:

pgdog/pgdog-config/src/otel.rs

Line 55 in e37160d

pub datadog_api_key: Option<String>,

[KennanHunter]

Great! I'll probably have an hour tonight to add that Datadog behavior, update and PR the documentation, and make any other changes you'd like to see.

I also wanted to quickly sneak in something to handle the case where these environment variables are defined but [otel] isn't present in the configuration, which currently disables any exporting even if the program has an OTEL_EXPORTER_ENDPOINT, which I know tripped me up. One way we could do that is to add a warning prompting them to add [otel] to their configuration, or we could change the behavior to use OTEL if the exporter is enabled via the env var, even when [otel] is not present. What are your thoughts?

Final thing: Would you be interested in this graphana_prometheus example? I could clean it (currently largely the work of Claude) and throw up a PR. I mostly implemented it for the sweet dopamine release of seeing that locked percent gauge redline.

[levkk]

I also wanted to quickly sneak in something to handle the case where these environment variables are defined but [otel] isn't present in the configuration, which currently disables any exporting even if the program has an OTEL_EXPORTER_ENDPOINT, which I know tripped me up. One way we could do that is to add a warning prompting them to add [otel] to their configuration, or we could change the behavior to use OTEL if the exporter is enabled via the env var, even when [otel] is not present. What are your thoughts?

I think you should be able to make it a not-Option in pgdog-config/core.rs and give it a Default implementation which fetches stuff from the env. If nothing in the env, flag it as disabled, or something like that?

Final thing: Would you be interested in this graphana_prometheus example? I could clean it (currently largely the work of Claude) and throw up a PR. I mostly implemented it for the sweet dopamine release of seeing that locked percent gauge redline.

Yes definitely. More examples = better!

[...] to add that Datadog behavior,

I need to test this manually against our DD account. I can do it quick, it's not an issue. The OTEL exporter was developed against datadog (it was me, my friend Claude and a DD dashboard), so I want to double check it still works there.

[levkk]

Sorry for delay here. The day got away from me a bit. I still need to double check this works against Datadog. 👍

[KennanHunter]

Useful info from https://docs.datadoghq.com/opentelemetry/guide/otlp_delta_temporality/?tab=python#implications-of-using-cumulative-aggregation-temporality:

If you opt to send OTLP monotonic sums, histograms, or exponential histograms with cumulative aggregation temporality, Datadog takes the difference between consecutive points on a timeseries. This means that:

• Your deployment is stateful, so you need to send all points on a timeseries to the same Datadog Agent or Datadog exporter. This affects how you scale your OpenTelemetry Collector deployments.
• Datadog might not send the first point it receives from a given timeseries if it cannot ensure this point is the true start of the timeseries. This may lead to missing points upon restarts.
• The minimum and maximum cannot be recovered for cumulative OTLP Histograms; they may be missing or approximated depending on the histograms export mode.

[KennanHunter]

Trying a new pattern. Let me know how you like it. If you do, we could in the future look at pulling this out and using it in other configuration spots if that proves useful.

For every [otel] field, we check both sources. The env var wins over the TOML value. If both are set and differ, it logs a warn! naming the field and the two values, then uses the env value.

[otel].temporality_preference resolution order (first match wins):

1. Env var OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE.
2. Explicit value in the [otel] TOML section.
3. Delta when datadog_api_key is set (via TOML or DD_API_KEY).
4. Cumulative otherwise.

If we set cumulative explicitly but have datadog's api key present it gives a pretty detailed warning as to why that's a bad idea, silence-able with a env var incase anyone finds a reason to want that behavior.

Take your time getting that datadog integration tested. When this does get merged, I'll throw up the PR for the grafana example just to keep history simple.