Endless "fail to ping" error message

[davidshen84]

Hi,

I installed the "valkey-operator" using "v0.0.59-chart" helm chart. Then I created a valkey instance. Everything worked fine. Recently, I noticed the following errors are flooding my system log.

The IP referenced in the message doesn't exist in my cluster. Meanwhile, I have a few services that rely on the Valkey instance, and they seem to be working fine.

It looks like the valkey-operator is trying to reach some dead instance and refuses to give up.

2025-08-02T05:02:39Z    INFO    upserting configmap     {"controller": "valkey", "controllerGroup": "hyperspike.io", "controllerKind": "Valkey", "Valkey": {"name":"valkey","namespace":"default"}, "namespace": "default", "name": "valkey", "reconcileID": "bd0a6844-c2a8-44bb-afa2-159c0f430dbf"}
2025-08-02T05:02:39Z    INFO    upserting service       {"controller": "valkey", "controllerGroup": "hyperspike.io", "controllerKind": "Valkey", "Valkey": {"name":"valkey","namespace":"default"}, "namespace": "default", "name": "valkey", "reconcileID": "bd0a6844-c2a8-44bb-afa2-159c0f430dbf"}
2025-08-02T05:02:39Z    INFO    upserting service       {"controller": "valkey", "controllerGroup": "hyperspike.io", "controllerKind": "Valkey", "Valkey": {"name":"valkey","namespace":"default"}, "namespace": "default", "name": "valkey", "reconcileID": "bd0a6844-c2a8-44bb-afa2-159c0f430dbf"}
2025-08-02T05:02:39Z    INFO    upserting service account       {"controller": "valkey", "controllerGroup": "hyperspike.io", "controllerKind": "Valkey", "Valkey": {"name":"valkey","namespace":"default"}, "namespace": "default", "name": "valkey", "reconcileID": "bd0a6844-c2a8-44bb-afa2-159c0f430dbf"}
2025-08-02T05:02:39Z    INFO    upserting pod disruption budget {"controller": "valkey", "controllerGroup": "hyperspike.io", "controllerKind": "Valkey", "Valkey": {"name":"valkey","namespace":"default"}, "namespace": "default", "name": "valkey", "reconcileID": "bd0a6844-c2a8-44bb-afa2-159c0f430dbf"}
2025-08-02T05:02:39Z    INFO    upserting statefulset   {"controller": "valkey", "controllerGroup": "hyperspike.io", "controllerKind": "Valkey", "Valkey": {"name":"valkey","namespace":"default"}, "namespace": "default", "name": "valkey", "reconcileID": "bd0a6844-c2a8-44bb-afa2-159c0f430dbf"}
2025-08-02T05:02:39Z    INFO    initializing cluster    {"controller": "valkey", "controllerGroup": "hyperspike.io", "controllerKind": "Valkey", "Valkey": {"name":"valkey","namespace":"default"}, "namespace": "default", "name": "valkey", "reconcileID": "bd0a6844-c2a8-44bb-afa2-159c0f430dbf"}
2025-08-02T05:02:39Z    INFO    setting epoch   {"controller": "valkey", "controllerGroup": "hyperspike.io", "controllerKind": "Valkey", "Valkey": {"name":"valkey","namespace":"default"}, "namespace": "default", "name": "valkey", "reconcileID": "bd0a6844-c2a8-44bb-afa2-159c0f430dbf", "pod": "valkey-0.valkey-headless.default.svc"}
2025-08-02T05:02:39Z    DEBUG   events  Setting epoch on pod valkey-0.valkey-headless.default.svc for default/valkey    {"type": "Normal", "object": {"kind":"Valkey","namespace":"default","name":"valkey","uid":"eca5aa6a-a15e-414f-aaa2-373d11a58c33","apiVersion":"hyperspike.io/v1","resourceVersion":"70781113"}, "reason": "Setting"}
2025-08-02T05:02:39Z    INFO    epoch already set       {"controller": "valkey", "controllerGroup": "hyperspike.io", "controllerKind": "Valkey", "Valkey": {"name":"valkey","namespace":"default"}, "namespace": "default", "name": "valkey", "reconcileID": "bd0a6844-c2a8-44bb-afa2-159c0f430dbf", "epoch": 1}
2025-08-02T05:02:39Z    INFO    setting slotrange       {"controller": "valkey", "controllerGroup": "hyperspike.io", "controllerKind": "Valkey", "Valkey": {"name":"valkey","namespace":"default"}, "namespace": "default", "name": "valkey", "reconcileID": "bd0a6844-c2a8-44bb-afa2-159c0f430dbf", "shard": 0}
2025-08-02T05:02:39Z    DEBUG   events  Setting slotrange on shard 0 for default/valkey {"type": "Normal", "object": {"kind":"Valkey","namespace":"default","name":"valkey","uid":"eca5aa6a-a15e-414f-aaa2-373d11a58c33","apiVersion":"hyperspike.io/v1","resourceVersion":"70781113"}, "reason": "Setting"}
2025-08-02T05:02:39Z    INFO    slotrange already set   {"controller": "valkey", "controllerGroup": "hyperspike.io", "controllerKind": "Valkey", "Valkey": {"name":"valkey","namespace":"default"}, "namespace": "default", "name": "valkey", "reconcileID": "bd0a6844-c2a8-44bb-afa2-159c0f430dbf"}
2025-08-02T05:02:42Z    ERROR   failed to ping valkey   {"controller": "valkey", "controllerGroup": "hyperspike.io", "controllerKind": "Valkey", "Valkey": {"name":"valkey","namespace":"default"}, "namespace": "default", "name": "valkey", "reconcileID": "bd0a6844-c2a8-44bb-afa2-159c0f430dbf", "error": "dial tcp 10.42.0.224:6379: connect: no route to host"}

[davidshen84]

I suspect the error is triggered by https://github.com/hyperspike/valkey-operator/blob/main/internal/controller/valkey_controller.go#L1569

It tries to ping an address defined at

valkey-operator/internal/controller/valkey_controller.go

Line 1533 in 88cd80a

InitAddress: []string{valkey.Name + "-0." + valkey.Name + "-headless." + valkey.Namespace + ".svc:6379"},

		InitAddress: []string{valkey.Name + "-0." + valkey.Name + "-headless." + valkey.Namespace + ".svc:6379"},

Is it trying to format a pod name or a service name?

There's no valkey-0.valkey-headless.default.svc in my system.

[davidshen84]

I think the root cause is how the operator tries to reach the other pods using the shorthand domain name. It seem it is no longer a recommended practice. I have tried many containers with nslookup installed, only the nslookup in the dnsutils container does the search operation, all other instances won't do the domain name search if there are more than one dot in the domain name.