External labels hide metrics on the remote read endpoint

[kkohtaka]

What did you do?

I tried the remote read API with the following master and slave configurations.

master (listening localhost:9090):

global:
  external_labels:
    role: master

remote_read:
- url: http://localhost:9091/api/v1/read

scrape_configs:
- job_name: prometheus
  static_configs:
  - targets:
    - localhost:9090

slave (listening localhost:9091):

global:
  external_labels:
    role: slave

scrape_configs:
- job_name: prometheus
  static_configs:
  - targets:
    - localhost:9091

Then, I ran a PromQL on the master Prometheus.

What did you expect to see?

We should get metrics ingested by both the master and the slave.

What did you see instead? Under which circumstances?

By running a query such as up{job="prometheus"}, I got only metrics ingested by the master.

$ curl -s 'http://localhost:9090/api/v1/query?query=up\{job="prometheus"\}' | jq .data.result[].metric
{
  "__name__": "up",
  "instance": "localhost:9090",
  "job": "prometheus"
}

NOTE: If we stop using the external labels on master Prometheus, the query shows both metrics by the master and the slave.

global:
  # external_labels:
  #   role: master

remote_read:
- url: http://localhost:9091/api/v1/read

scrape_configs:
- job_name: prometheus
  static_configs:
  - targets:
    - localhost:9090

$ curl -s 'http://localhost:9090/api/v1/query?query=up\{job="prometheus"\}' | jq .data.result[].metric
{
  "__name__": "up",
  "instance": "localhost:9090",
  "job": "prometheus"
}
{
  "__name__": "up",
  "instance": "localhost:9091",
  "job": "prometheus",
  "role": "slave"
}

Environment

• System information:

  Darwin 16.7.0 x86_64

• Prometheus version:

  prometheus, version 2.0.0-rc.2 (branch: master, revision: 7aa2930)
  build user: kkohtaka@sirius.local
  build date: 20171026-14:49:58
  go version: go1.8.3

[kkohtaka]

Matchers for external labels are always added to a remote read query. Is that a correct behaviour?

https://github.com/prometheus/prometheus/blob/master/storage/remote/read.go#L70

[brian-brazil]

This behaviour is correct and intended, as the data in the slave Prometheus does not have a role=master label and thus does not match the query up.

For this sort of setup to work your slaves need to have the same external labels as the master, plus an extra label to distinguish the slaves.

[kkohtaka]

Thanks, I confirmed that I can fetch both metrics from the master and the slave by using the following configurations.

master:

global:
  external_labels:
    cluster: foo
...

slave:

global:
  external_labels:
    cluster: foo
    slave: 'true'
...

$ curl -s 'http://localhost:9090/api/v1/query?query=up\{job="prometheus"\}' | jq .data.result[].metric     
{
  "__name__": "up",
  "instance": "localhost:9090",
  "job": "prometheus"
}
{
  "__name__": "up",
  "instance": "localhost:9091",
  "job": "prometheus",
  "slave": "true"
}

I understand the current behaviour and design but I don't know if they are the best. Adding the same external labels as the master to the slave seems unnecessarily large requirement because external labels are not only used for remote endpoints but also federations and alertmanagers.

[brian-brazil]

slave: 'true'

This isn't quite right. You need a different value per slave. If you only have one slave, then you shouldn't be using remote read - rather merge the slave into the master.

Adding the same external labels as the master to the slave seems unnecessarily large requirement because external labels are not only used for remote endpoints but also federations and alertmanagers.

That is the whole point of external labels. They uniquely identify the Prometheus server, so data from different Prometheus servers doesn't get mixed up.

[kkohtaka]

slave: 'true'

This isn't quite right. You need a different value per slave. If you only have one slave, then you shouldn't be using remote read - rather merge the slave into the master.

Aha. I will choose a different value per slave in an actual environment.

Adding the same external labels as the master to the slave seems unnecessarily large requirement because external labels are not only used for remote endpoints but also federations and alertmanagers.

That is the whole point of external labels. They uniquely identify the Prometheus server, so data from different Prometheus servers doesn't get mixed up.

OK. I understand that we should use a set of external labels as an identifier of Prometheus instance.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.