External Labels Overwrite existing Labels in Thanos

[ebini]

Hi,

i’m using a prometheus federation installation which scrapes the data from 3 Prometheus instances. (Don't ask why, this is at the moment necessary)

External Labels are:

Prometheus Federate:
app=„federate“
env=„prod“

Prometheus 1
app=„abc“
env=„dev“

Prometheus 2
app=„abc“
env=„test“

Prometheus 3
app=„abc“
env=„prod“

In the Prometheus Federation my queries locks fine:
for example on Prometheus (Federate) the query
mymetric{env=„prod“}
returns the correct metric
mymetric{app=„ abc“,env=„prod“} = 0

But on Thanos Query Gateway the Announced LabelSets of the Prometheus Federate is app=„federate“ and env=„prod“.

In it seems to overwrite the existing Labels in the metrics
So if i do the same query:
mymetric{env=„prod“}
i now get 3 metrics, all with the same labels:
mymetric{app=„federate“,env=„prod“} = 0
mymetric{app=„federate“,env=„prod“} = 0
mymetric{app=„federate“,env=„prod“} = 0

It seems Thanos „overwrites“ the existing Labels with the accounced ones.
I don't know if this is a bug, or i have done something wrong.
Perhaps you can help
(Note, at the moment i have to scrape a prometheus federation installation and can not change to prometheus with thanos sidecar installed)

Thanks in advance

[krasi-georgiev]

Maybe try playing with the dedup labels in the querier.

https://github.com/thanos-io/thanos/blob/master/docs/components/query.md#deduplication

[GiedriusS]

Hi, perhaps you need to specify --query.replica-label=app on Thanos Query?

[ebini]

Thanks, but it doesn't change anything.
It still seems that the direkt query to Prometheus isn't the same than using Thanos.

And i don't have duplicate data in the Prometheus instances. These are not redundant prometheus instances.

[MarcMielke]

same problem, looks like the ingesting prometheus instance wins and overwrites the external label(s) already set in the prometheus instances being scraped from the federation endpoint. honor-labels is ignored as it seems. here is what I tried (where cluster is the external label):

    - job_name: monitoring/gke-preprod
      honor_timestamps: true
      honor_labels: true
      scrape_interval: 60s
      scrape_timeout: 30s
      metrics_path: /federate
      scheme: http
      params:
        match[]:
        - '{__name__=~".+"}'
      static_configs:
      - targets:
        - xx.xx.xx.xx:9090
        labels:
          cluster: gke-preprod

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[QianChenglong]

This problem is a bit annoying, hopefully it can be fixed.

[stale]

This issue/PR has been automatically marked as stale because it has not had recent activity. Please comment on status otherwise the issue will be closed in a week. Thank you for your contributions.

[ashleyprimo]

Hi 👋 - has anyone found a fix for this?