Metrics didn't scrape from new targets

[stamm]

What did you do?

I use forked version of prometheus 1.3.0 with support discovery from etcd: https://github.com/THE108/prometheus

I update keys in etcd for new targets. Old targets is removed and new target appears in prometheus targets list. But it stay with status "Unknown" in about 10 minutes. This time may vary. After status is changed to "Up".
Usually after that it works normally.
But sometimes it has last scrape like this "11m12.642s ago" and it's grow for a while (maybe several minutes) without getting metrics from that targets. After it works ok.

What did you expect to see?
That target has status "UP"

What did you see instead? Under which circumstances?
See status Unknown and status "UP" with big last scrape

Environment

• System information:
  Linux 3.10.0-327.10.1.el7.x86_64 x86_64

• Prometheus version:

prometheus, version 1.3.0 (branch: HEAD, revision: c887ed97b0f8f8d748de3aa6a11b20b7ad522e2b)
build user: root@runner-b1ef57c1-project-858-concurrent-0
build date: 20161102-09:53:10
go version: go1.7.1

• Prometheus configuration file:

global: 
  scrape_interval: "10s"
  scrape_timeout: "9s"
  evaluation_interval: "15s"
rule_files: 
  - /etc/prometheus/generated-rules.rule
scrape_configs: 
  - job_name: prometheus
    static_configs: 
      - targets: 
          - "localhost:9090"
  - job_name: goapi
    scrape_interval: "20s"
    etcd_sd_configs: 
      - endpoints: 
          - "http://etcd1:2379"
          - "http://etcd2:2379"
          - "http://etcd3:2379"
          - "http://etcd4:2379"
          - "http://etcd5:2379"
        key_prefix: /live
…

[brian-brazil]

I'd presume by default the issue is with the forked code. If you can reproduce this on a vanilla Prometheus please let us know.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.