Remote read doesn't read between 1.8.2 and 2.0

[theonlydoo]

What did you do?
Try to remote read from prometheus 2.0 on 1.8.2 instance

What did you expect to see?
Metrics
What did you see instead? Under which circumstances?
No metrics
Environment

• System information:

  Linux 4.9.0-3-amd64 x86_64

• Prometheus version:

prometheus, version 1.8.2 (branch: HEAD, revision: 5211b96d4d1291c3dd1a569f711d3b301b635ecb)
  build user:       root@1412e937e4ad
  build date:       20171104-16:09:14
  go version:       go1.9.2

prometheus, version 2.0.0 (branch: HEAD, revision: 0a74f98628a0463dddc90528220c94de5032d1a0)
  build user:       root@615b82cb36b6
  build date:       20171108-07:11:59
  go version:       go1.9.2

• Prometheus configuration file:

• prometheus 2.0

# my global config
global:
  scrape_interval:     15s # By default, scrape targets every 15 seconds.
  evaluation_interval: 15s
  scrape_timeout:       15s
  # scrape_timeout is set to the global default (10s).

  # Attach these labels to any time series or alerts when communicating with
  # external systems (federation, remote storage, Alertmanager).
  external_labels:
    region: fr
    monitor: 'prod'
    env: prod
remote_read:
    - url: "http://oldserver.tld:9090/api/v1/read"
      read_recent: false
      remote_timeout: 30s
# Load and evaluate rules in this file every 'evaluation_interval' seconds.
rule_files:
  - "/etc/prometheus/rules/somerulefile"
  - "/etc/prometheus/rules/somerulefile"
  - "/etc/prometheus/rules/somerulefile"
  - "/etc/prometheus/rules/somerulefile"
  - "/etc/prometheus/rules/somerulefile"
  - "/etc/prometheus/rules/somerulefile"

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: somejobname

    # Override the global default and scrape targets from this job every 5 seconds.
    scrape_interval: 5s

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

    static_configs:
      - targets: ['127.0.0.1:9090']


### Infrastructure autoconf ###

scrape_configs:
  - job_name: somejobname
    scrape_interval: 30s
    file_sd_configs:
      - files:
        - /etc/prometheus/config/somefile
  - job_name: somejobname
    scrape_interval: 30s
    file_sd_configs:
      - files:
        - /etc/prometheus/config/somefile
  - job_name: somejobname
    scrape_interval: 30s
    file_sd_configs:
      - files:
        - /etc/prometheus/config/somefile

• prometheus 1.8.2 :

# my global config
global:
  scrape_interval:     15s # By default, scrape targets every 15 seconds.
  evaluation_interval: 15s
  scrape_timeout:       15s
  # scrape_timeout is set to the global default (10s).

  # Attach these labels to any time series or alerts when communicating with
  # external systems (federation, remote storage, Alertmanager).
  external_labels:
    region: fr
    monitor: 'prod'
    env: prod

# Load and evaluate rules in this file every 'evaluation_interval' seconds.
rule_files:
  - "/etc/prometheus/rules/somerulefile"
  - "/etc/prometheus/rules/somerulefile"
  - "/etc/prometheus/rules/somerulefile"
  - "/etc/prometheus/rules/somerulefile"
  - "/etc/prometheus/rules/somerulefile"
  - "/etc/prometheus/rules/somerulefile"

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: somejobname

    # Override the global default and scrape targets from this job every 5 seconds.
    scrape_interval: 5s

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

    static_configs:
      - targets: ['127.0.0.1:9090']


### Infrastructure autoconf ###

scrape_configs:

  - job_name: somejobname
    scrape_interval: 30s
    file_sd_configs:
      - files:
        - /etc/prometheus/config/somefile
        - /etc/prometheus/config/somefile
        - /etc/prometheus/config/somefile

  - job_name: somejobname
    scrape_interval: 2m
    file_sd_configs:
      - files:
        - /etc/prometheus/config/somefile
        - /etc/prometheus/config/somefile
        - /etc/prometheus/config/somefile
        - /etc/prometheus/config/somefile

  - job_name: somejobname
    scrape_interval: 30s
    file_sd_configs:
      - files:
        - /etc/prometheus/config/somefile

What have I done

I tried to let default settings on on remote_read, and custom them, read_recent has been set to true and
false with the exact same result

N.b. : scraping and rule evaluation are done correctly and have no problem whatsoever.

[tomwilkie]

• Do you get any errors in the log or in the UI for the 2.0 server?
• What queries have you run? Have you tried something simple (like up)?

I highly doubt this is a bug, I suspect this is misconfiguration, and therefore probably belongs on the mailing list.

[theonlydoo]

indeed, it was a misconfig

[pavelnemirovsky]

What was the issue? I see exactly same behavior

[theonlydoo]

a pebkac, probably, restarted both configs, works now :

remote_read:
    - url: "http://myoldprom:9090/api/v1/read"
      read_recent: false
      remote_timeout: 30s

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.