Tempo (v2.5.0) Distributor missing tempo_receiver_accepted_spans

[jameshounshell]

Describe the bug
Today I upgraded the tempo-distributed helm chart from 1.8.5 to 1.10.0 (app version 2.3.1 to 2.5.0) and I noticed in the Tempo Operational dashboard, that the tempo_receiver_accepted_spans metric disappeared. I'm assuming this the result to a change in the embedded OTLP library but other than the change noted in the release, there was nothing specific about this metric, making me think breaking it was unintentional.

To Reproduce
Steps to reproduce the behavior:

1. Install version 1.10.0 of the tempo-distributed helm chart.
2. Verify the absence of tempo_receiver_accepted_spans at /metrics of the tempo distributor component.

Expected behavior
I would expect the metric tempo_receiver_accepted_spans to still exist/be-emitted.

Environment:

• Infrastructure: GKE fully managed
• Deployment tool: Kustomize plus helm (helmCharts) chart

Additional Context
Here is a graph showing tempo_receiver_ metrics disappearing while tempo_distributor_spans_received_total remains.

[joe-elliott]

Thank you for the issue. This is likely due to an upgrade of OTEL:

#3515
#3148

I would recommend using the metric you pointed out tempo_distributor_spans_received_total. I would like to restore this metric, if someone wants to attempt a PR.

[akasarapu]

I'm also facing the same issue, tempo_receiver_accepted_spans and tempo_receiver_refused_spans metrics are missing after I have upgraded my tempo chart to 1.10.0 from 1.9.1 and image to 2.5.0 from 2.4.1.

@joe-elliott/ @jameshounshell, Any idea on if this issue fixed or still exists?. Or any work around we have to do to make it work?

[joe-elliott]

It still exists. We would need to find a way to extract that metric again from the otel receivers to publish it.

[KrishnaJyothika]

Hi @joe-elliott, Any update on this? Is there any way we can get tempo_receiver_accepted_spans and tempo_receiver_refused_spans metrics in 2.5.0v?

[joe-elliott]

We will need a PR to re-expose that internal otel metric. Please take a look if you have time.