path based routing for signals causing issue with /v1/metrics

[kadhamecha-conga]

Hi Team,

I’m experimenting with FQDN endpoints and path-based routing via NGINX for OpenTelemetry signals (/v1/traces, /v1/metrics, /v1/logs). Each path is routed to separate backend pods. centralized otel collector uses prometheusremotewrite to push data in mimir.

Traces and logs are working as expected, but I’m encountering an issue with metrics:

Metrics requests are failing with HTTP 500 errors
Errors are marked as non-retryable: “Exporting failed. The error is not retryable. Dropping data.”
OTEL clients (especially DaemonSets) start dropping metrics after retry attempts

Has anyone seen this behavior or have suggestions on what might be causing it?

[s3onghyun]

The not retryable. Dropping data. line is the key: it's your client-side OTLP/HTTP exporter reacting to the status code it got back on /v1/metrics. Per the OTLP/HTTP spec, only a small set of codes are retryable — in the collector's otlphttpexporter, isRetryableStatusCode returns true only for 429, 502, 503, 504. A 500 is treated as a permanent error, so the exporter doesn't retry and drops the batch. That behavior is correct; the real question is why /v1/metrics returns 500 while /v1/traces and /v1/logs don't.

First, a topology note that narrows it fast. A collector exports asynchronously by default (the exporter sending_queue is enabled), so the receiving collector's OTLP receiver enqueues the batch and returns 200 before the prometheusremotewrite → Mimir write happens. That means a Mimir rejection normally shows up as an error in the receiving collector's logs, not as a 500 back to your client. So a 500 actually reaching your DaemonSets points more at NGINX or the receiving collector itself than at Mimir — unless you've disabled/overflowed the sending queue (block_on_overflow), which makes backpressure propagate to the receiver.

Isolate the hop that emits the 500:

1. Bypass NGINX — POST an OTLP metrics payload straight to the metrics backend collector's :4318/v1/metrics. If that succeeds, the 500 is from NGINX (path rewrite/upstream config); if it 500s, it's the collector.
2. Check the metrics backend collector logs for exporter errors around the failing requests.
3. Check the Mimir distributor response codes for the corresponding pushes.

Most common causes specific to the metrics path (PRW → Mimir), i.e. what traces/logs don't hit:

• Missing tenant header. If Mimir multi-tenancy is on, the prometheusremotewrite exporter must send X-Scope-OrgID; without it Mimir rejects the write. Add it under the exporter's headers:. (Traces/logs going to Tempo/Loki have their own tenant config, which is why only metrics break.)
• Delta temporality. prometheusremotewrite requires cumulative temporality. If your SDKs emit delta, the translation fails — run the deltatocumulative processor first, or set the SDK temporality preference to cumulative.
• Out-of-order / duplicate samples. Multiple DaemonSet pods producing the same series (identical or missing identifying resource attributes) make Mimir reject with out-of-order/duplicate errors. Make sure each source carries distinguishing resource attributes that end up as labels.

My guess given the symptom (a 500 that actually reaches and is dropped by the client, only on metrics) is NGINX routing on /v1/metrics or the receiving collector failing synchronously — step 1 will tell you in one request. If instead the backend logs show Mimir 4xx, it's one of the three PRW causes above.