Prometheus connection refused to target node-exporter which is running on Openstack VM

[yobshans]

Hi,
I faced with an issue while evaluate monitoring approach with Prometheus/cAdvisor/Node-exporter on Openstack Overcloud Instance/VM.
Actually I have deployed successfully cAdvisor/Node-exporter on all nodes.
But, installed Node-exporter container on Openstack Instance(VM) is not reachable by Prometheus
Error: Get http://10.0.0.107:9100/metrics: dial tcp 10.0.0.107:9100: connect: connection refused
However, I can successfully run commands on Prometheus server such as:
curl http://10.0.0.107:9100/metrics
wget http://10.0.0.107:9100/metrics
telnet 172.0.1.14 9100
nmap 10.0.0.107
Starting Nmap 6.40 ( http://nmap.org ) at 2018-07-25 18:47 IDT
Nmap scan report for 10.0.0.107
Host is up (0.00078s latency).
Not shown: 998 filtered ports
PORT STATE SERVICE
22/tcp open ssh
9100/tcp open jetdirect
MAC Address: FA:16:3E:58:C3:1E (Unknown)
Nmap done: 1 IP address (1 host up) scanned in 4.47 seconds

More details:
Openstack Instance/VM based on RHEL 7 image
cat /etc/system-release
Red Hat Enterprise Linux Server release 7.5 (Maipo)
Prometheus Version | 2.3.2
71af5e2
HEAD
root@5258e0bd9cc1
20180712-14:02:52
go1.10.3

Docker version
lient:
Version: 1.13.1
API version: 1.26
Package version: docker-1.13.1-68.gitdded712.el7.x86_64
Go version: go1.9.2
Git commit: dded712/1.13.1
Built: Tue Jun 12 18:30:09 2018
OS/Arch: linux/amd64
Server:
Version: 1.13.1
API version: 1.26 (minimum version 1.12)
Package version: docker-1.13.1-68.gitdded712.el7.x86_64
Go version: go1.9.2
Git commit: dded712/1.13.1
Built: Tue Jun 12 18:30:09 2018
OS/Arch: linux/amd64
Experimental: false
docker inspect node-exporter
[
{
"Id": "cc3d589c475d093857e143cb3606ae1d9b60c35c951e8d4a6f3f318e1d573167",
"Created": "2018-07-24T16:28:45.399133007Z",
"Path": "/bin/node_exporter",
"Args": [
"--path.procfs=/host/proc",
"--path.sysfs=/host/sys",
"--collector.filesystem.ignored-mount-points",
"^/(sys|proc|dev|host|etc|rootfs/var/lib/docker/containers|rootfs/var/lib/docker/overlay2|rootfs/run/docker/netns|rootfs/var/lib/docker/aufs)(9687|/)",
"--no-collector.hwmon"
],
"State": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 18690,
"ExitCode": 0,
"Error": "",
"StartedAt": "2018-07-24T16:28:45.604712989Z",
"FinishedAt": "0001-01-01T00:00:00Z"
},

Configuration:

job_name: 3-Instances
scrape_interval: 15s
scrape_timeout: 10s
metrics_path: /metrics
scheme: http
static_configs:
    targets:
        10.0.0.107:9100
        labels:
        name: cloud-1-1-vm-node-exporter
    targets:
        10.0.0.107:9191
        labels:
        name: cloud-1-1-vm-cadvisor

Thank you

Proposal

Use case. Why is this important?

Nice to have' is not a good use case :)

Bug Report

What did you do?

What did you expect to see?

What did you see instead? Under which circumstances?

Environment

• System information:

  insert output of uname -srm here

• Prometheus version:

  insert output of prometheus --version here

• Alertmanager version:

  insert output of alertmanager --version here (if relevant to the issue)

• Prometheus configuration file:

insert configuration here

• Alertmanager configuration file:

insert configuration here (if relevant to the issue)

• Logs:

insert Prometheus and Alertmanager logs relevant to the issue here

[yobshans]

Screenshot

[simonpasquier]

Thanks for your report. It looks as if this is actually a question about usage.

To make your question, and all replies, easier to find, we suggest you move this over to our user mailing list, which you can also search. If you prefer more interactive help, join or our IRC channel, #prometheus on irc.freenode.net. Please be aware that our IRC channel has no logs, is not searchable, and that people might not answer quickly if they are busy or asleep. If in doubt, you should choose the mailing list.

Once your questions have been answered, please add a short line pointing to relevant replies in case anyone stumbles here via a search engine in the future.

[simonpasquier]

We have debugged it offline and the problem was the network.
Running the Prometheus container with "--network=host" solved the issue.

[yobshans]

I really appreciate for your help.
Thanks a lot for quick and concrete assistance.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.