remote_read from prometheus1.8.2 doesn't work

[zerthimon]

Bug Report

What did you do?
I'm in the process of migration from 1.8.2 to 2.6.1

Old container 1.8.2:
started with --web.listen-address=:9090

global:
  external_labels:
    monitor: prometheus-old

New container 2.6.1:

remote_read:
  - url: http://prometheus-server-old:9090/api/v1/read

Both docker containers are on the same host, the new container links the old one on the name prometheus-server-old

What did you expect to see?
Metrics from the old container.

What did you see instead? Under which circumstances?
Only metrics from the the new container.
I'm able to get metrics inside the new container from the old container using curl http://prometheus-server-old:9090/api/v1/query?query= ...

[cstyan]

Can you show us logs from the 2.6.1 server? When you say you only see metrics from the 2.6.1 container, you mean when you run a query? To be clear, remote read would run queries against your other container, it doesn't mean the 2.6.1 instance is actively reading the data from your old prometheus instance and storing it locally.

[zerthimon]

@cstyan yeah I'm querying 2.6.1 for metrics that exist on 1.8.2 and don't get any.
I'm using the same queries (grafana dashboards) i used with 1.8.2. The backend is now 2.6.1 with remote read pointing to 1.8.2 container.
I don't see any logs relevant to remote reads, just the TSDB operations.

[bboreham]

EDIT: I was wrong, sorry.

I might be missing something, but I am under the impression no version of Prometheus supports remote_read or remote_write as a server, only as a client.

The remote interface was added to allow talking to a non-Prometheus store, e.g. Cortex or Influxdb.

[brian-brazil]

There's a remote read endpoint on Prometheus itself.