[prometheusexporter] Expose native histograms

[krisztianfekete]

Component(s)

exporter/prometheus

What happened?

Description

Currently, prometheusreceiver supports native histograms, but prometheusexporter cannot expose them.

Steps to Reproduce

1. Instrument your code with native histograms
2. Scrape the metrics with prometheusreceiver by following the instructions here: https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/f5d54da37ae664e1a0b6934a61d38f1afab5501f/receiver/prometheusreceiver/README.md#prometheus-native-histograms

Expected Result

prometheusexporter should expose native histograms

Actual Result

Logs via debugexporter:

{"level":"error","ts":1718965637.50062,"caller":"prometheusexporter@v0.103.0/accumulator.go:94","msg":"failed to translate metric","kind":"exporter","data_type":"metrics","name":"prometheus","data_type":"\u0004","metric_name":"gloo_mesh_translation_time_sec","stacktrace":"github.com/open-telemetry/opentelemetry-collector-contrib/exporter/prometheusexporter.(*lastValueAccumulator).addMetric
\tgithub.com/open-telemetry/opentelemetry-collector-contrib/exporter/prometheusexporter@v0.103.0/accumulator.go:94
github.com/open-telemetry/opentelemetry-collector-contrib/exporter/prometheusexporter.(*lastValueAccumulator).Accumulate
\tgithub.com/open-telemetry/opentelemetry-collector-contrib/exporter/prometheusexporter@v0.103.0/accumulator.go:71
github.com/open-telemetry/opentelemetry-collector-contrib/exporter/prometheusexporter.(*collector).processMetrics
\tgithub.com/open-telemetry/opentelemetry-collector-contrib/exporter/prometheusexporter@v0.103.0/collector.go:88
github.com/open-telemetry/opentelemetry-collector-contrib/exporter/prometheusexporter.(*prometheusExporter).ConsumeMetrics
\tgithub.com/open-telemetry/opentelemetry-collector-contrib/exporter/prometheusexporter@v0.103.0/prometheus.go:85
go.opentelemetry.io/collector/exporter/exporterhelper.(*metricsRequest).Export
\tgo.opentelemetry.io/collector/exporter@v0.103.0/exporterhelper/metrics.go:59
go.opentelemetry.io/collector/exporter/exporterhelper.(*timeoutSender).send
\tgo.opentelemetry.io/collector/exporter@v0.103.0/exporterhelper/timeout_sender.go:49
go.opentelemetry.io/collector/exporter/exporterhelper.(*baseRequestSender).send
\tgo.opentelemetry.io/collector/exporter@v0.103.0/exporterhelper/common.go:37
go.opentelemetry.io/collector/exporter/exporterhelper.(*metricsSenderWithObservability).send
\tgo.opentelemetry.io/collector/exporter@v0.103.0/exporterhelper/metrics.go:158
--

Collector version

v0.103.0

Environment information

N/A

OpenTelemetry Collector configuration

receivers:
  prometheus:
    config:
      global:
        evaluation_interval: 1m
        scrape_interval: 15s
        scrape_timeout: 15s
        scrape_protocols: [ PrometheusProto, OpenMetricsText1.0.0, OpenMetricsText0.0.1, PrometheusText0.0.4 ]
        scrape_configs:
          <your targets exposing native histograms>
          ...
pipelines:
  metrics/test:
    receivers:
    - prometheus
    exporters:
    - debug
    - prometheus

Log output

{"level":"error","ts":1718965637.50062,"caller":"prometheusexporter@v0.103.0/accumulator.go:94","msg":"failed to translate metric","kind":"exporter","data_type":"metrics","name":"prometheus","data_type":"\u0004","metric_name":"gloo_mesh_translation_time_sec","stacktrace":"github.com/open-telemetry/opentelemetry-collector-contrib/exporter/prometheusexporter.(*lastValueAccumulator).addMetric
\tgithub.com/open-telemetry/opentelemetry-collector-contrib/exporter/prometheusexporter@v0.103.0/accumulator.go:94
github.com/open-telemetry/opentelemetry-collector-contrib/exporter/prometheusexporter.(*lastValueAccumulator).Accumulate
\tgithub.com/open-telemetry/opentelemetry-collector-contrib/exporter/prometheusexporter@v0.103.0/accumulator.go:71
github.com/open-telemetry/opentelemetry-collector-contrib/exporter/prometheusexporter.(*collector).processMetrics
\tgithub.com/open-telemetry/opentelemetry-collector-contrib/exporter/prometheusexporter@v0.103.0/collector.go:88
github.com/open-telemetry/opentelemetry-collector-contrib/exporter/prometheusexporter.(*prometheusExporter).ConsumeMetrics
\tgithub.com/open-telemetry/opentelemetry-collector-contrib/exporter/prometheusexporter@v0.103.0/prometheus.go:85
go.opentelemetry.io/collector/exporter/exporterhelper.(*metricsRequest).Export
\tgo.opentelemetry.io/collector/exporter@v0.103.0/exporterhelper/metrics.go:59
go.opentelemetry.io/collector/exporter/exporterhelper.(*timeoutSender).send
\tgo.opentelemetry.io/collector/exporter@v0.103.0/exporterhelper/timeout_sender.go:49
go.opentelemetry.io/collector/exporter/exporterhelper.(*baseRequestSender).send
\tgo.opentelemetry.io/collector/exporter@v0.103.0/exporterhelper/common.go:37
go.opentelemetry.io/collector/exporter/exporterhelper.(*metricsSenderWithObservability).send
\tgo.opentelemetry.io/collector/exporter@v0.103.0/exporterhelper/metrics.go:158
--

Additional context

No response

[github-actions]

Pinging code owners:

• exporter/prometheus: @Aneurysm9

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[wildum]

Hi, I tried it with the following config, and the native histograms are exposed just like they are exposed by my app:

receivers:
    prometheus:
      config:
        scrape_configs:
          - job_name: 'otel-collector'
            scrape_interval: 5s
            static_configs:
              - targets: ['localhost:9001']

exporters:
  prometheus:
    endpoint: "localhost:9010"
    const_labels:
      test_name: v1

service:
  pipelines:
    metrics:
      receivers: [prometheus]
      exporters: [prometheus]
  telemetry:
    logs:
      level: debug

Native histogram metrics exposed by my app:
golang_native_histogram_bucket{address="0.0.0.0",port="9001",le="+Inf"} 1523
golang_native_histogram_sum{address="0.0.0.0",port="9001"} 773369.1551833635
golang_native_histogram_count{address="0.0.0.0",port="9001"} 1523

Native histogram metrics exposed by the prometheus exporter:
golang_native_histogram_bucket{address="0.0.0.0",instance="localhost:9001",job="otel-collector",port="9001",test_name="v1",le="+Inf"} 1284
golang_native_histogram_sum{address="0.0.0.0",instance="localhost:9001",job="otel-collector",port="9001",test_name="v1"} 656156.1718255163
golang_native_histogram_count{address="0.0.0.0",instance="localhost:9001",job="otel-collector",port="9001",test_name="v1"} 1284

Could you share your full setup?

[krisztianfekete]

Are you sure these are native histograms? They look like regular ones to me. The required options are seemingly missing from your receiver config: https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/receiver/prometheusreceiver/README.md#prometheus-native-histograms, so my guess is that you're falling back to the legacy ones here.

EDIT: I added them to the original post, so it's easier to reproduce.

[wildum]

Ah you are right, I was missing the scape_protocols, I can reproduce it now, thanks :)