"Context deadline exceeded" when using host names in Vagrant+Landrush

[cubranic]

What did you do?

I set up a multi-node Vagrant, using the Landrush plugin to provide dynamic name resolution of VMs on both host and guests. This works fine in that curl and other utilities I've tried resolve the names correctly. But when I use those names as targets in prometheus.yml, the targets are shown as "down" with error "context deadline exceeded". If I use their IP address, Prometheus is able to scrape them just fine.

What did you expect to see?

Successful polling of the targets regardless of whether they are specified by host name or IP.

What did you see instead? Under which circumstances?

Using the hostname, I get no stats with "context deadline exceeded" error message.

Environment

VirtualBox 5.1.12 on 10.12.2
Vagrant 1.8.7
Landrush 1.2.0

• System information:

  Linux 3.10.0-327.el7.x86_64 x86_64

• Prometheus version:

  prometheus, version 1.4.1 (branch: master, revision: 2a89e87)
  build user: root@e685d23d8809
  build date: 20161128-09:59:22
  go version: go1.7.3

• Prometheus configuration file:

# my global config
global:
  scrape_interval:     15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
  external_labels:
      monitor: 'codelab-monitor'

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'node'
    static_configs:
      - targets:
        - 'node0:9100'
        - 'node1:9100'
        - 'node2:9100'

• Vagrantfile:

Vagrant.configure("2") do |config|

  config.vm.box = "puppetlabs/centos-7.2-64-nocm"

  config.vm.network "private_network", type: 'dhcp'
  config.landrush.enabled = true

  (0..2).reverse_each do |i|
    config.vm.define 'node' + i.to_s, primary: (i==0) do |node|
      node.vm.hostname = 'node' + i.to_s + '.vagrant.test'
    end
  end
end

[brian-brazil]

This is most likely an issue with your resolver setup, I'd suggest debugging from there.

[cubranic]

Doesn't my resolver work just fine if curl node1:9100/metrics works correctly? Ditto for ping, ssh, etc.

[brian-brazil]

A resolver works fine, there's more than one way to do resolution on a machine though. Have you checked what's in your resolv.conf?

[cubranic]

# Generated by NetworkManager
search wireless.ubc.ca vagrant.test
nameserver 10.0.2.3

[cubranic]

It looks like it works if I use the FQDN:

  - job_name: 'node'
    static_configs:
      - targets:
        - 'node0.vagrant.test:9100'
        - 'node1.vagrant.test:9100'
        - 'node2.vagrant.test:9100'

But I don't have to use those outside Prometheus, because vagrant.test is on the resolver's domain search list.

[brian-brazil]

Not all resolvers support search paths.

[cubranic]

Can you please expand on that -- is Prometheus using a different resolver from every other executable on this system? Because everything else works fine with the short name.

[brian-brazil]

We're moving all support-related questions to https://groups.google.com/forum/#!forum/prometheus-users if you're still having this issue.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.