Matching labels with remote_read does not work

[lgogolin]

Bug Report

What did you do?
Having prometheus-proxy with remote_read configuration that read data from two other prometheus.

What works ?

up{project="monitoring",region="us-east-1"}
up{project="monitoring",region="eu-central-1"}
up{project="monitoring",region=~".*"}

What does not work?

up{project="monitoring",region=~"eu-central-1"}
up{project="monitoring",region=~"us-east-1"}
up{project="monitoring",region=~"us-east-1|eu-central-1"}

Environment

• Prometheus version:

prometheus, version 2.4.3 (branch: HEAD, revision: 167a4b4e73a8eca8df648d2d2043e21bdb9a7449)
  build user:       root@1e42b46043e9
  build date:       20181004-08:42:02
  go version:       go1.11.1

• Prometheus configuration file:
  prometheus-proxy

remote_read:
        # monitoring - eu-central-1
        - url: http://prometheus-eu-central-1:9090/api/v1/read
          read_recent: true
          required_matchers:
            project: monitoring
        # # monitoring - us-eas-1
        - url: http://prometheus-us-east-1/api/v1/read
          read_recent: true
          required_matchers:
            project: monitoring

prometheus-eu-central-1:

global:
        external_labels:
          project: monitoring
          region: "eu-central-1"
          env: production

scrape_configs:
  ...

prometheus us-east-1:

global:
        external_labels:
          project: monitoring
          region: "us-east-1"
          env: production

scrape_configs:
  ...

[brian-brazil]

The issue here is that the data inside the readee has no region label, it's only removed/added by the remote read logic and that can only deal with exact matches.

Federation is probably a better choice here.

[lgogolin]

Case is, if I add more than one label for required_matchers it stops working completely.

remote_read:
        # monitoring - eu-central-1
        - url: http://prometheus-eu-central-1:9090/api/v1/read
          read_recent: true
          required_matchers:
            project: monitoring
            region: "eu-central-1"
        # # monitoring - us-eas-1
        - url: http://prometheus-us-east-1/api/v1/read
          read_recent: true
          required_matchers:
            project: monitoring
            region: "us-east-1"

And according to documentation it suppose to support list of matchers:

# An optional list of equality matchers which have to be
# present in a selector to query the remote read endpoint.
required_matchers:
  [ <labelname>: <labelvalue> ... ]

[lgogolin]

It's even weirder. After I restart POD with prometheus-proxy, first query works:

up{project="monitoring",region="us-east-1"}

But trying next even with other label, like:

up{project="monitoring",region="eu-central-1"}

is not returning data anymore

[lgogolin]

Any update regarding this bug ?