Prometheus unable to resolve DNS to send alert notifications

[drewandersen]

What did you do?
Created two virtual machines on my local box for development: one running Prometheus and one running Alertmanager. Purposely triggered an alert on Prometheus.

What did you expect to see?
The alert in Alertmanager.

What did you see instead? Under which circumstances?
No alert. Instead, the following error:

017-07-06_21:23:04.60141 time="2017-07-06T21:23:04Z" level=error msg="Error sending alerts: Post [alertmanager_dns]:80/api/v1/alerts: dial tcp: lookup [alertmanager_dns] on 127.0.0.1:53: no such host" alertmanager="[alertmanager_dns]:80/api/v1/alerts" count=1 source="notifier.go:379"

Environment
Docker or Virtualbox running Ubuntu 14.04.

• System information:
  Linux 4.8.0-56-generic x86_64

• Prometheus version:
  prometheus, version 1.7.1 (branch: master, revision: 3afb3ff)
  build user: root@0aa1b7fc430d
  build date: 20170612-11:44:05
  go version: go1.8.3

• Alertmanager version:
  alertmanager, version 0.5.1 (branch: master, revision: 0ea1cac51e6a620ec09d053f0484b97932b5c902)
  build user: root@fb407787b8bf
  build date: 20161125-08:14:40
  go version: go1.7.3

Additional Information
The alertmanager_dns in the above error message resolves to an alias for another DNS which contains the A record. Can curl and dig the alertmanager_dns from the box where Prometheus is running and receive the expected result. Can even use the Alertmanager API to trigger an alert from the box running Prometheus. The metric prometheus_notifications_alertmanagers_discovered shows a consistent value of 1. If I launch Prometheus and specify the DNS that contains the actual A record, the behavior is as expected (alerts are sent from Prometheus to Alertmanager). Also ran a quick Go service on the same host where Prometheus is running to make a GET request to alertmanager_dns, and it succeeded.

Essentially, the DNS is resolvable everywhere except where Prometheus attempts to send alerts to Alertmanager.

[drewandersen]

This is also true when using file_sd_configs in the scrape_configs section of the Prometheus config file.

[brian-brazil]

Usage questions are best asked on https://groups.google.com/forum/#!forum/prometheus-users

[drewandersen]

So it's expected behavior for Prometheus to detect an Alertmanager instance but fail to send alert messages?

[snguyen-ith]

@drewandersen Did you ever resolve this issue?

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.