metrics-generator stops working

[suikast42]

I am on the latest LTGM version on my dev env and on production.

Suddenly overnight metrics-generator stops to produce span metrics in production environment.

I have exactly the same conifigurations in dev and prod. ( The single diffrecne are the storage durations 1d -> 30 d ).

The whole setup of LGTM is with single deployment and local file storage.

I can't see any errors in tempo or mimir. The apps can push the traces and metrics but only the metrics-generator won't produce the metrics from the traces.

Development Env:

Production Env:

[joe-elliott]

This is a dupe of #2514. Can you post your details there?

[suikast42]

Close in favour of #2514