Ping: Timeout reading from socket

[simonszu]

Hi,

i have defined a ping-module which is basically just an icmp prober. I am probing around 17 hosts with it. Now i have a problem: The probe_success for one of this host is 0, although the host is pingable when executing ping in my shell.
Systemd journal shows the following error:

Nov 10 14:58:21 tirn blackbox_exporter[29216]: time="2016-11-10T14:58:21Z" level=info msg="Timeout reading from socket for grandstream: read ip4 0.0.0.0: i/o timeout" source="icmp.go:93"

The blackbox_exporter binary is run as root and has CAP_NET_RAW permissions. All other ping probes succeed, and since pinging this host succeeds as well via the shell, i think this could be blackbox_exporter's fault - couldn't it?

[brian-brazil]

Are you sure your test pings are using icmp?

A tcpdump of the working and failing pings would be useful.

[hasso]

Might be the problem with resolving hostname. Go programs generally use the resolver written in go and which sends DNS requests directly to the servers defined in /etc/resolv.conf. Ping command from CLI uses libc getnameinfo(3) etc functions. So, if the name is resolvable using /etc/hosts file in the system, but not via nameservers, you might have a problem with similar symptoms.

See "Name Resolution" section from https://golang.org/pkg/net/ for more info.

[simonszu]

Both types of host are resolvable via nameservers, @hasso
So i made a tcpdump. This is the cli output of blackbox_exporter pinging the failing host:

11:04:16.286345 IP tirn.home.simonszu.de > grandstream.home.simonszu.de: ICMP echo request, id 29216, seq 57166, length 36
11:04:16.286707 IP grandstream.home.simonszu.de > tirn.home.simonszu.de: ICMP echo reply, id 29216, seq 57166, length 36

And this is the cli output of blackbox_exporter pinging a successful host:

11:05:41.002031 IP tirn.home.simonszu.de > yamaha.home.simonszu.de: ICMP echo request, id 29216, seq 57260, length 36
11:05:41.002238 IP yamaha.home.simonszu.de > tirn.home.simonszu.de: ICMP echo reply, id 29216, seq 57260, length 36

I will try to attach the dumped output - no privacy concerns since the only communication between the blackbox_exporter host and the targets are the packets generated by blackbox_exporter itself.

Note: The blackbox_exporter host is 192.168.1.2, the successful host is the .10 and the failing host is the .5

Maybe you have an idea?

tcpdumps.zip

[brian-brazil]

The packets from failing box have the wrong checksums, and are presumably being dropped by the kernel. I expect something weird is going on in the network stack on that box.

[simonszu]

This wouldn't surprise me. It is a quite old VoIP phone i bought ten years ago on ebay. It works, but i noticed the first quirks going on when i started to use icinga to monitor my network. The phone has a builtin web server for config, and the web server probe went to critical everytime someone actually used the phone for calling.
Strangely, the blackbox_exporter probe does not alert a failing web server when someone calls on this phone - so, yeah, i think i just have to accept the weirdness.
Thanks anyway.