ISAR ODFNodeLatencyHighOnOSDNodes

[rbo]

Description	Active since	State	Value
Node 10.32.105.3 has max ICMP RTT latency > 10ms over the last 24h. This may impact Ceph performance.	
Apr 16, 2026, 2:12 AM
 Firing	1.7593969e-02
Node 10.32.105.5 has max ICMP RTT latency > 10ms over the last 24h. This may impact Ceph performance.	
Apr 16, 2026, 1:04 AM
 Firing	2.0138177e-02
Node 10.32.105.6 has max ICMP RTT latency > 10ms over the last 24h. This may impact Ceph performance.	
Apr 15, 2026, 9:32 PM
 Firing	1.0729407e-02

[rbo]

Ping storm6 -> storm5

oc debug node/storm6
Starting pod/storm6-debug-pgld8 ...
To use host binaries, run `chroot /host`. Instead, if you need to access host namespaces, run `nsenter -a -t 1`.
Pod IP: 10.32.105.6
If you don't see a command prompt, try pressing enter.
sh-5.1# ping 10.32.105.5
PING 10.32.105.5 (10.32.105.5) 56(84) bytes of data.
64 bytes from 10.32.105.5: icmp_seq=1 ttl=64 time=2.83 ms
64 bytes from 10.32.105.5: icmp_seq=2 ttl=64 time=0.175 ms
64 bytes from 10.32.105.5: icmp_seq=3 ttl=64 time=0.155 ms
64 bytes from 10.32.105.5: icmp_seq=4 ttl=64 time=0.185 ms
64 bytes from 10.32.105.5: icmp_seq=5 ttl=64 time=0.209 ms
^C
--- 10.32.105.5 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4075ms
rtt min/avg/max/mdev = 0.155/0.710/2.830/1.059 ms
sh-5.1#

[rbo]

oc get pods -o wide -l app.kubernetes.io/name=odf-blackbox-exporter
NAME                                     READY   STATUS    RESTARTS   AGE   IP             NODE     NOMINATED NODE   READINESS GATES
odf-blackbox-exporter-5cb5864897-r7s2z   1/1     Running   0          8d    10.128.16.59   storm5   <none>           <none>

[rbo]

oc logs  -l app.kubernetes.io/name=odf-blackbox-exporter
time=2026-04-07T18:50:42.179Z level=WARN source=icmp.go:345 msg="Timeout reading from socket" module=icmp_internal target=10.32.105.6 err="read udp 0.0.0.0:1424: raw-read udp 0.0.0.0:1424: i/o timeout"
time=2026-04-07T18:50:42.180Z level=ERROR source=handler.go:130 msg="Probe failed" module=icmp_internal target=10.32.105.6 duration_seconds=4.500318216
time=2026-04-07T18:51:11.047Z level=WARN source=icmp.go:345 msg="Timeout reading from socket" module=icmp_internal target=10.32.105.6 err="read udp 0.0.0.0:1429: raw-read udp 0.0.0.0:1429: i/o timeout"
time=2026-04-07T18:51:11.047Z level=ERROR source=handler.go:130 msg="Probe failed" module=icmp_internal target=10.32.105.6 duration_seconds=4.500892897
time=2026-04-07T18:51:12.180Z level=WARN source=icmp.go:345 msg="Timeout reading from socket" module=icmp_internal target=10.32.105.6 err="read udp 0.0.0.0:1430: raw-read udp 0.0.0.0:1430: i/o timeout"
time=2026-04-07T18:51:12.181Z level=ERROR source=handler.go:130 msg="Probe failed" module=icmp_internal target=10.32.105.6 duration_seconds=4.500887526
time=2026-04-07T18:51:41.048Z level=WARN source=icmp.go:345 msg="Timeout reading from socket" module=icmp_internal target=10.32.105.6 err="read udp 0.0.0.0:1435: raw-read udp 0.0.0.0:1435: i/o timeout"
time=2026-04-07T18:51:41.049Z level=ERROR source=handler.go:130 msg="Probe failed" module=icmp_internal target=10.32.105.6 duration_seconds=4.502135697
time=2026-04-07T18:51:42.179Z level=WARN source=icmp.go:345 msg="Timeout reading from socket" module=icmp_internal target=10.32.105.6 err="read udp 0.0.0.0:1436: raw-read udp 0.0.0.0:1436: i/o timeout"
time=2026-04-07T18:51:42.180Z level=ERROR source=handler.go:130 msg="Probe failed" module=icmp_internal target=10.32.105.6 duration_seconds=4.500188683

[rbo]

Let's restart the blackbox exporter...

 oc delete pods  -l app.kubernetes.io/name=odf-blackbox-exporter
pod "odf-blackbox-exporter-5cb5864897-r7s2z" deleted