[Bug]: java.net.UnknownHostException: my-cluster-zookeeper-0.my-cluster-zookeeper-nodes.default.svc

[tonyliu666]

Bug Description

My environment is kubaedm cluster and one master, three worker nodes. I follow the basic installation listed in this link,https://strimzi.io/quickstarts/, but I can't create the pod,my-cluster-kafka-0. And the custom resource I created is followed by this command, kubectl apply -f https://strimzi.io/examples/latest/kafka/kafka-ephemeral.yaml -n kafka. The partial logs coming from the operator pod are like these:

2024-04-30 13:32:21 ERROR StaticHostProvider:148 - Unable to resolve address: my-cluster-zookeeper-0.my-cluster-zookeeper-nodes.kafka.svc/:2181
java.net.UnknownHostException: my-cluster-zookeeper-0.my-cluster-zookeeper-nodes.kafka.svc: Name or service not known
at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method) ~[?:?]
at java.net.InetAddress$PlatformNameService.lookupAllHostAddr(InetAddress.java:934) ~[?:?]
at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1543) ~[?:?]
at java.net.InetAddress$NameServiceAddresses.get(InetAddress.java:852) ~[?:?]
at java.net.InetAddress.getAllByName0(InetAddress.java:1533) ~[?:?]
at java.net.InetAddress.getAllByName(InetAddress.java:1385) ~[?:?]
at java.net.InetAddress.getAllByName(InetAddress.java:1306) ~[?:?]
at org.apache.zookeeper.client.StaticHostProvider$1.getAllByName(StaticHostProvider.java:88) ~[org.apache.zookeeper.zookeeper-3.8.3.jar:3.8.3]
at org.apache.zookeeper.client.StaticHostProvider.resolve(StaticHostProvider.java:141) ~[org.apache.zookeeper.zookeeper-3.8.3.jar:3.8.3]
at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:368) ~[org.apache.zookeeper.zookeeper-3.8.3.jar:3.8.3]
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1204) ~[org.apache.zookeeper.zookeeper-3.8.3.jar:3.8.3]
2024-04-30 13:32:21 WARN ClientCnxnSocketNetty:151 - future isn't success.

But I can access this service hostname by deploying the dnsutil pod on the kafka namespace. Here are the outputs from the testing results:

k exec -ti dnsutils nslookup my-cluster-zookeeper-0.my-cluster-zookeeper-nodes.kafka.svc
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
Server: 10.96.0.10
Address: 10.96.0.10#53

Name: my-cluster-zookeeper-0.my-cluster-zookeeper-nodes.kafka.svc.cluster.local
Address: 10.85.0.38

I don't know why the service has been existed in the kafka namespace, but the zookeeper pod, my-cluster-zookeeper-0, my-cluster-zookeeper-1 and my-cluster-zookeeper-2 can't access it.

Steps to reproduce

just follow the link,https://strimzi.io/quickstarts/. But deploying the different custom resource setup with this command, kubectl apply -f https://strimzi.io/examples/latest/kafka/kafka-ephemeral.yaml -n kafka.

Expected behavior

Get the my-cluster-kafka-0 outputs from entering kubectl get pods -n kafka

Strimzi version

0.40.0

Kubernetes version

kubernetes v1.28.0

Installation method

yaml files

Infrastructure

kubeadm

Configuration files and logs

By the way the partial logs from zookeeper are like these,

2024-04-30 13:40:30,139 WARN Cannot open channel to 3 at election address my-cluster-zookeeper-2.my-cluster-zookeeper-nodes.kafka.svc/10.85.0.66:3888 (org.apache.zookeeper.server.quorum.QuorumCnxManager) [QuorumConnectionThread-[myid=1]-29]
java.net.NoRouteToHostException: No route to host
at java.base/sun.nio.ch.Net.pollConnect(Native Method)
at java.base/sun.nio.ch.Net.pollConnectNow(Net.java:672)
at java.base/sun.nio.ch.NioSocketImpl.timedFinishConnect(NioSocketImpl.java:554)
at java.base/sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:602)
at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:327)
at java.base/java.net.Socket.connect(Socket.java:633)
at java.base/sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:304)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.initiateConnection(QuorumCnxManager.java:384)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$QuorumConnectionReqThread.run(QuorumCnxManager.java:458)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:840)

Additional context

No response

[scholzj]

It is not really clear where do you get the different errors. So I guess a key would be to share the full logs or better explain what is what and from where it is.

Unable to resolve address: my-cluster-zookeeper-0.my-cluster-zookeeper-nodes.kafka.svc/:2181 or No route to host suggest networking / DNS issues. So that would be something you can look into.

[tonyliu666]

OK,currently I have these pods are on kafka namespace.

k get pods
NAME READY STATUS RESTARTS AGE
dnsutils 1/1 Running 0 32h
my-cluster-zookeeper-0 1/1 Running 0 35m
my-cluster-zookeeper-1 1/1 Running 0 35m
my-cluster-zookeeper-2 1/1 Running 0 35m
nginx 1/1 Running 0 29h
strimzi-cluster-operator-86df856f74-258k9 1/1 Running 0 38m

The first error logs listed on the bug description are in strimzi-cluster-operator-86df856f74-258k9 this pod, and the error logs listed on Configuration files and logs are in my-cluster-zookeeper-0 this pod.

Here are the files corresponding to the whole logs recorded in my-cluster-zookeeper-0 and strimzi-cluster-operator-86df856f74-258k9.
strimzi-cluster.log
zookeeper.log

[scholzj]

Yeah, the logs simply look like some networking / DNS issues. Please keep in mind that issues like that can be limited to a particular worker node as well. So what might seem to work for you on one worker or on master might not work on another worker.

[tonyliu666]

Ok,got it.Thanks for your helps.I will try to fix it.