Prometheus Stack on Rancher : Error refreshing DNS targets

[dbortnic]

What did you do?
Hello everybody.
I deployed the Prometheus stack from Rancher community templates.
So it is supposed to work out of the box.

The only thing i changed was adding a job to the "/prom-conf/prometheus.yml" in the "prometheus" container and increasing intervals everywhere it was possible.
I checked the "/targets" in the prometheus web-ui and the job i added was "up"
I created a data source in Grafana and imported a few Dashboard for Docker but most of graphs say "no data point". I noticed that the metrics of the job i added does not contain the queries used by graphs.

I checked the "prometheus" container logs and saw the following :

1/18/2018 4:30:04 PMWARN[0992] DNS resolution failed. name=node-exporter reason=read udp 10.42.234.39:60991->169.254.169.250:53: i/o timeout server=169.254.169.250 source=dns.go:188
1/18/2018 4:30:04 PMWARN[0992] DNS resolution failed. name=cadvisor.prometheus.rancher.internal reason=read udp 10.42.234.39:34635->169.254.169.250:53: i/o timeout server=169.254.169.250 source=dns.go:188
1/18/2018 4:30:04 PMWARN[0992] DNS resolution failed. name=prometheus-rancher-exporter reason=read udp 10.42.234.39:57178->169.254.169.250:53: i/o timeout server=169.254.169.250 source=dns.go:188
1/18/2018 4:30:06 PMWARN[0994] DNS resolution failed. name=node-exporter reason=read udp 10.42.234.39:33317->169.254.169.250:53: i/o timeout server=169.254.169.250 source=dns.go:188
1/18/2018 4:30:06 PMWARN[0994] DNS resolution failed. name=cadvisor.prometheus.rancher.internal reason=read udp 10.42.234.39:48666->169.254.169.250:53: i/o timeout server=169.254.169.250 source=dns.go:188
1/18/2018 4:30:06 PMWARN[0994] DNS resolution failed. name=prometheus-rancher-exporter reason=read udp 10.42.234.39:51780->169.254.169.250:53: i/o timeout server=169.254.169.250 source=dns.go:188
1/18/2018 4:30:08 PMWARN[0996] DNS resolution failed. name=node-exporter reason=read udp 10.42.234.39:41043->169.254.169.250:53: i/o timeout server=169.254.169.250 source=dns.go:188
1/18/2018 4:30:08 PMWARN[0996] DNS resolution failed. name=cadvisor.prometheus.rancher.internal reason=read udp 10.42.234.39:57478->169.254.169.250:53: i/o timeout server=169.254.169.250 source=dns.go:188
1/18/2018 4:30:08 PMWARN[0996] DNS resolution failed. name=prometheus-rancher-exporter reason=read udp 10.42.234.39:43906->169.254.169.250:53: i/o timeout server=169.254.169.250 source=dns.go:188
1/18/2018 4:30:10 PMWARN[0998] DNS resolution failed. name=node-exporter reason=read udp 10.42.234.39:58135->169.254.169.250:53: i/o timeout server=169.254.169.250 source=dns.go:188
1/18/2018 4:30:10 PMERRO[0998] Error refreshing DNS targets: could not resolve node-exporter: no server responded source=dns.go:115
1/18/2018 4:30:10 PMWARN[0998] DNS resolution failed. name=prometheus-rancher-exporter reason=read udp 10.42.234.39:48976->169.254.169.250:53: i/o timeout server=169.254.169.250 source=dns.go:188
1/18/2018 4:30:10 PMERRO[0998] Error refreshing DNS targets: could not resolve prometheus-rancher-exporter: no server responded source=dns.go:115
1/18/2018 4:30:10 PMWARN[0998] DNS resolution failed. name=cadvisor.prometheus.rancher.internal reason=read udp 10.42.234.39:42383->169.254.169.250:53: i/o timeout server=169.254.169.250 source=dns.go:188
1/18/2018 4:30:10 PMERRO[0998] Error refreshing DNS targets: could not resolve cadvisor.prometheus.rancher.internal: no server responded source=dns.go:115
1/18/2018 4:30:34 PMWARN[1022] DNS resolution failed. name=node-exporter reason=read udp 10.42.234.39:58764->169.254.169.250:53: i/o timeout server=169.254.169.250 source=dns.go:188
1/18/2018 4:30:34 PMWARN[1022] DNS resolution failed. name=cadvisor.prometheus.rancher.internal reason=read udp 10.42.234.39:47946->169.254.169.250:53: i/o timeout server=169.254.169.250 source=dns.go:188
1/18/2018 4:30:34 PMWARN[1022] DNS resolution failed. name=prometheus-rancher-exporter reason=read udp 10.42.234.39:56043->169.254.169.250:53: i/o timeout server=169.254.169.250 source=dns.go:188
1/18/2018 4:30:36 PMWARN[1024] DNS resolution failed. name=node-exporter reason=read udp 10.42.234.39:44318->169.254.169.250:53: i/o timeout server=169.254.169.250 source=dns.go:188
1/18/2018 4:30:36 PMWARN[1024] DNS resolution failed. name=cadvisor.prometheus.rancher.internal reason=read udp 10.42.234.39:45087->169.254.169.250:53: i/o timeout server=169.254.169.250 source=dns.go:188
1/18/2018 4:30:36 PMWARN[1024] DNS resolution failed. name=prometheus-rancher-exporter reason=read udp 10.42.234.39:47490->169.254.169.250:53: i/o timeout server=169.254.169.250 source=dns.go:188
1/18/2018 4:30:38 PMWARN[1026] DNS resolution failed. name=node-exporter reason=read udp 10.42.234.39:50738->169.254.169.250:53: i/o timeout server=169.254.169.250 source=dns.go:188
1/18/2018 4:30:38 PMWARN[1026] DNS resolution failed. name=cadvisor.prometheus.rancher.internal reason=read udp 10.42.234.39:52479->169.254.169.250:53: i/o timeout server=169.254.169.250 source=dns.go:188
1/18/2018 4:30:38 PMWARN[1026] DNS resolution failed. name=prometheus-rancher-exporter reason=read udp 10.42.234.39:50088->169.254.169.250:53: i/o timeout server=169.254.169.250 source=dns.go:188
1/18/2018 4:30:40 PMWARN[1028] DNS resolution failed. name=node-exporter reason=read udp 10.42.234.39:59056->169.254.169.250:53: i/o timeout server=169.254.169.250 source=dns.go:188

Environment
Docker CE - 17.06.2-ce
Rancher Server - 1.6.9
Prometheus Template version - 3.0

• Prometheus configuration file:

global:
  scrape_interval:     25s
  evaluation_interval: 25s
  external_labels:
      monitor: 'exporter-metrics'

scrape_configs:

- job_name: 'HostsMetrics'
  dns_sd_configs:
  - names:
    - node-exporter
    refresh_interval: 30s
    type: A
    port: 9100

- job_name: 'ContainerMetrics'
  static_configs:
    - targets:
      - 'rancher-server:9108'

- job_name: 'Docker'  #this is the job i added
  static_configs:
    - targets:
      - '10.1.1.228:9323'

- job_name: 'RancherServerMetrics'
  dns_sd_configs:
  - names:
    - cadvisor.prometheus.rancher.internal
    refresh_interval: 30s
    type: SRV
    port: 8080

- job_name: 'RancherApi'
  dns_sd_configs:
  - names:
    - 'prometheus-rancher-exporter'
    refresh_interval: 30s
    type: A
    port: 9173

- job_name: 'Prometheus'
  static_configs:
    - targets:
       - '127.0.0.1:9090'

I have the feeling i`m missing something very basic, please tell me if any information is needed.
Thank you.

[dbortnic]

UPDATE.
I launched the stack on another Rancher server with 4 hosts (the first had 1 host) and it worked "as is". Still don`t understand what is the cause for that "DNS" error on the first Rancher server.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.