Zookeeper not able to form cluster in OKE (Oracle Kubernetes Cluster)

[rmampuzha-aie]

Kubernetes version v1.29.1
Operator version [0.40.0]
Installed with 3 replicas.

helm install strimzi-operator strimzi/strimzi-kafka-operator -n kafka-operator-ns -f values_ha.yaml
kubectl apply -f kafka-ha.yaml -n kafka-ns

Error
2024-05-02 20:10:12,429 WARN Cannot open channel to 1 at election address kafka-cluster-zookeeper-0.kafka-cluster-zookeeper-nodes.kafka-ns.svc/:3888 (org.apache.zookeeper.server.quorum.QuorumCnxManager) [QuorumConnectionThread-[myid=2]-15]
java.net.UnknownHostException: kafka-cluster-zookeeper-0.kafka-cluster-zookeeper-nodes.kafka-ns.svc
at java.base/sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:572)
at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:327)
at java.base/java.net.Socket.connect(Socket.java:633)
at java.base/sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:304)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.initiateConnection(QuorumCnxManager.java:384)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$QuorumConnectionReqThread.run(QuorumCnxManager.java:458)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:840)

All zookeeper PODs has same error.

values_ha.yaml

watchNamespaces:
  - kafka-ns
affinity:
  nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      nodeSelectorTerms:
        - matchExpressions:
          - key: name
            operator: In
            values:
            - kafka-node-pool
    preferredDuringSchedulingIgnoredDuringExecution:
    - weight: 1
      preference:
        matchExpressions:
        - key: app
          operator: In
          values:
          - Kafka

kafka-ha.yaml

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: kafka-cluster
spec:
  kafka:
    version: 3.7.0
    replicas: 3
    listeners:
      - name: plain
        port: 9092
        type: internal
        tls: false
      - name: tls
        port: 9093
        type: internal
        tls: true
    readinessProbe:
      initialDelaySeconds: 15
      timeoutSeconds: 5
    livenessProbe:
      initialDelaySeconds: 15
      timeoutSeconds: 5
    template:
      pod:
        affinity:
          nodeAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              nodeSelectorTerms:
                - matchExpressions:
                  - key: name
                    operator: In
                    values:
                    - kafka-node-pool
            preferredDuringSchedulingIgnoredDuringExecution:
            - weight: 1
              preference:
                matchExpressions:
                - key: app
                  operator: In
                  values:
                  - Kafka
#       tolerations:
#       - key: "workload"
#         operator: "Equal"
#         value: "Kafka"
#         effect: "NoSchedule"
        metadata:
          annotations:
            prometheus.io/scrape: "true"
            prometheus.io/path: /metrics
            prometheus.io/port: "9404"
    config:
      offsets.topic.replication.factor: 3
      transaction.state.log.replication.factor: 3
      transaction.state.log.min.isr: 2
      inter.broker.protocol.version: "3.5"
      min.insync.replicas: 2
      default.replication.factor: 2
      auto.create.topics.enable: "true"
      delete.topic.enable: "true"
      log.retention.hours: 48
      num.partitions: 18
    storage:
      type: jbod
      volumes:
      - id: 0
        type: persistent-claim
        size: 50Gi
        deleteClaim: false
        class: "oci"
    metricsConfig:
      type: jmxPrometheusExporter
      valueFrom:
        configMapKeyRef:
          name: kafka-metrics
          key: kafka-metrics-config.yml
    jvmOptions:
      -Xms: 3g
      -Xmx: 3g
    resources:
      requests:
        memory: 2Gi
        cpu: "500m"
      limits:
        memory: 4Gi
  zookeeper:
    replicas: 3
    resources:
      requests:
        memory: 1Gi
        cpu: "300m"
      limits:
        memory: 4Gi
    jvmOptions:
      -Xms: 2048m
      -Xmx: 2048m
    template:
      pod:
        affinity:
          nodeAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              nodeSelectorTerms:
                - matchExpressions:
                  - key: name
                    operator: In
                    values:
                    - kafka-node-pool
            preferredDuringSchedulingIgnoredDuringExecution:
            - weight: 1
              preference:
                matchExpressions:
                - key: app
                  operator: In
                  values:
                  - Kafka
#        tolerations:
#        - key: "workload"
#          operator: "Equal"
#          value: "Kafka"
#          effect: "NoSchedule"
        metadata:
          annotations:
            prometheus.io/scrape: "true"
            prometheus.io/path: /metrics
            prometheus.io/port: "9404"
    readinessProbe:
      initialDelaySeconds: 15
      timeoutSeconds: 5
    livenessProbe:
      initialDelaySeconds: 15
      timeoutSeconds: 5
    storage:
      type: persistent-claim
      size: 50Gi
      class: "oci"
      deleteClaim: false
    metricsConfig:
      type: jmxPrometheusExporter
      valueFrom:
        configMapKeyRef:
          name: kafka-metrics
          key: zookeeper-metrics-config.yml
  entityOperator:
    topicOperator: {}
    userOperator: {}
  kafkaExporter:
    topicRegex: ".*"
    groupRegex: ".*"
---
kind: ConfigMap
apiVersion: v1
metadata:
  name: kafka-metrics
  labels:
    app: strimzi
data:
  kafka-metrics-config.yml: |
    # See https://github.com/prometheus/jmx_exporter for more info about JMX Prometheus Exporter metrics
    lowercaseOutputName: true
    rules:
    # Special cases and very specific rules
    - pattern: kafka.server<type=(.+), name=(.+), clientId=(.+), topic=(.+), partition=(.*)><>Value
      name: kafka_server_$1_$2
      type: GAUGE
      labels:
       clientId: "$3"
       topic: "$4"
       partition: "$5"
    - pattern: kafka.server<type=(.+), name=(.+), clientId=(.+), brokerHost=(.+), brokerPort=(.+)><>Value
      name: kafka_server_$1_$2
      type: GAUGE
      labels:
       clientId: "$3"
       broker: "$4:$5"
    - pattern: kafka.server<type=(.+), cipher=(.+), protocol=(.+), listener=(.+), networkProcessor=(.+)><>connections
      name: kafka_server_$1_connections_tls_info
      type: GAUGE
      labels:
        cipher: "$2"
        protocol: "$3"
        listener: "$4"
        networkProcessor: "$5"
    - pattern: kafka.server<type=(.+), clientSoftwareName=(.+), clientSoftwareVersion=(.+), listener=(.+), networkProcessor=(.+)><>connections
      name: kafka_server_$1_connections_software
      type: GAUGE
      labels:
        clientSoftwareName: "$2"
        clientSoftwareVersion: "$3"
        listener: "$4"
        networkProcessor: "$5"
    - pattern: "kafka.server<type=(.+), listener=(.+), networkProcessor=(.+)><>(.+):"
      name: kafka_server_$1_$4
      type: GAUGE
      labels:
       listener: "$2"
       networkProcessor: "$3"
    - pattern: kafka.server<type=(.+), listener=(.+), networkProcessor=(.+)><>(.+)
      name: kafka_server_$1_$4
      type: GAUGE
      labels:
       listener: "$2"
       networkProcessor: "$3"
    # Some percent metrics use MeanRate attribute
    # Ex) kafka.server<type=(KafkaRequestHandlerPool), name=(RequestHandlerAvgIdlePercent)><>MeanRate
    - pattern: kafka.(\w+)<type=(.+), name=(.+)Percent\w*><>MeanRate
      name: kafka_$1_$2_$3_percent
      type: GAUGE
    # Generic gauges for percents
    - pattern: kafka.(\w+)<type=(.+), name=(.+)Percent\w*><>Value
      name: kafka_$1_$2_$3_percent
      type: GAUGE
    - pattern: kafka.(\w+)<type=(.+), name=(.+)Percent\w*, (.+)=(.+)><>Value
      name: kafka_$1_$2_$3_percent
      type: GAUGE
      labels:
        "$4": "$5"
    # Generic per-second counters with 0-2 key/value pairs
    - pattern: kafka.(\w+)<type=(.+), name=(.+)PerSec\w*, (.+)=(.+), (.+)=(.+)><>Count
      name: kafka_$1_$2_$3_total
      type: COUNTER
      labels:
        "$4": "$5"
        "$6": "$7"
    - pattern: kafka.(\w+)<type=(.+), name=(.+)PerSec\w*, (.+)=(.+)><>Count
      name: kafka_$1_$2_$3_total
      type: COUNTER
      labels:
        "$4": "$5"
    - pattern: kafka.(\w+)<type=(.+), name=(.+)PerSec\w*><>Count
      name: kafka_$1_$2_$3_total
      type: COUNTER
    # Generic gauges with 0-2 key/value pairs
    - pattern: kafka.(\w+)<type=(.+), name=(.+), (.+)=(.+), (.+)=(.+)><>Value
      name: kafka_$1_$2_$3
      type: GAUGE
      labels:
        "$4": "$5"
        "$6": "$7"
    - pattern: kafka.(\w+)<type=(.+), name=(.+), (.+)=(.+)><>Value
      name: kafka_$1_$2_$3
      type: GAUGE
      labels:
        "$4": "$5"
    - pattern: kafka.(\w+)<type=(.+), name=(.+)><>Value
      name: kafka_$1_$2_$3
      type: GAUGE
    # Emulate Prometheus 'Summary' metrics for the exported 'Histogram's.
    # Note that these are missing the '_sum' metric!
    - pattern: kafka.(\w+)<type=(.+), name=(.+), (.+)=(.+), (.+)=(.+)><>Count
      name: kafka_$1_$2_$3_count
      type: COUNTER
      labels:
        "$4": "$5"
        "$6": "$7"
    - pattern: kafka.(\w+)<type=(.+), name=(.+), (.+)=(.*), (.+)=(.+)><>(\d+)thPercentile
      name: kafka_$1_$2_$3
      type: GAUGE
      labels:
        "$4": "$5"
        "$6": "$7"
        quantile: "0.$8"
    - pattern: kafka.(\w+)<type=(.+), name=(.+), (.+)=(.+)><>Count
      name: kafka_$1_$2_$3_count
      type: COUNTER
      labels:
        "$4": "$5"
    - pattern: kafka.(\w+)<type=(.+), name=(.+), (.+)=(.*)><>(\d+)thPercentile
      name: kafka_$1_$2_$3
      type: GAUGE
      labels:
        "$4": "$5"
        quantile: "0.$6"
    - pattern: kafka.(\w+)<type=(.+), name=(.+)><>Count
      name: kafka_$1_$2_$3_count
      type: COUNTER
    - pattern: kafka.(\w+)<type=(.+), name=(.+)><>(\d+)thPercentile
      name: kafka_$1_$2_$3
      type: GAUGE
      labels:
        quantile: "0.$4"
    # KRaft mode: uncomment the following lines to export KRaft related metrics
    # KRaft overall related metrics
    # distinguish between always increasing COUNTER (total and max) and variable GAUGE (all others) metrics
    #- pattern: "kafka.server<type=raft-metrics><>(.+-total|.+-max):"
    #  name: kafka_server_raftmetrics_$1
    #  type: COUNTER
    #- pattern: "kafka.server<type=raft-metrics><>(.+):"
    #  name: kafka_server_raftmetrics_$1
    #  type: GAUGE
    # KRaft "low level" channels related metrics
    # distinguish between always increasing COUNTER (total and max) and variable GAUGE (all others) metrics
    #- pattern: "kafka.server<type=raft-channel-metrics><>(.+-total|.+-max):"
    #  name: kafka_server_raftchannelmetrics_$1
    #  type: COUNTER
    #- pattern: "kafka.server<type=raft-channel-metrics><>(.+):"
    #  name: kafka_server_raftchannelmetrics_$1
    #  type: GAUGE
    # Broker metrics related to fetching metadata topic records in KRaft mode
    #- pattern: "kafka.server<type=broker-metadata-metrics><>(.+):"
    #  name: kafka_server_brokermetadatametrics_$1
    #  type: GAUGE
  zookeeper-metrics-config.yml: |
    # See https://github.com/prometheus/jmx_exporter for more info about JMX Prometheus Exporter metrics
    lowercaseOutputName: true
    rules:
    # replicated Zookeeper
    - pattern: "org.apache.ZooKeeperService<name0=ReplicatedServer_id(\\d+)><>(\\w+)"
      name: "zookeeper_$2"
      type: GAUGE
    - pattern: "org.apache.ZooKeeperService<name0=ReplicatedServer_id(\\d+), name1=replica.(\\d+)><>(\\w+)"
      name: "zookeeper_$3"
      type: GAUGE
      labels:
        replicaId: "$2"
    - pattern: "org.apache.ZooKeeperService<name0=ReplicatedServer_id(\\d+), name1=replica.(\\d+), name2=(\\w+)><>(Packets\\w+)"
      name: "zookeeper_$4"
      type: COUNTER
      labels:
        replicaId: "$2"
        memberType: "$3"
    - pattern: "org.apache.ZooKeeperService<name0=ReplicatedServer_id(\\d+), name1=replica.(\\d+), name2=(\\w+)><>(\\w+)"
      name: "zookeeper_$4"
      type: GAUGE
      labels:
        replicaId: "$2"
        memberType: "$3"
    - pattern: "org.apache.ZooKeeperService<name0=ReplicatedServer_id(\\d+), name1=replica.(\\d+), name2=(\\w+), name3=(\\w+)><>(\\w+)"
      name: "zookeeper_$4_$5"
      type: GAUGE
      labels:
        replicaId: "$2"
        memberType: "$3"

[scholzj]

The things you shared are unreadable because they are not formatted. But the UnknownHostException suggests this is some kind of DNS or networking issue.

[rmampuzha-aie]

I updated the yaml to make it readable. Yes the issue is they are unable to communicate. I am able to reach using IP of the POD. So it is something related to DNS. In OKE they use CoreDNS service. Not sure it is something to do with that. Anyway this same configuration used to work in GKE.

[scholzj]

CoreDNS is used on most Kubernetes cluster. But it can be misconfigured, or slow, it can be some issue on a particular worker node etc.

[rmampuzha-aie]

There was an issue with CoreDNS on the nodes. It has been resolved now. However the issue still remains. Below logs are from Operator

2024-05-04 03:23:22 ERROR StaticHostProvider:148 - Unable to resolve address: kafka-cluster-zookeeper-2.kafka-cluster-zookeeper-nodes.kafka-ns.svc/<unresolved>:2181
java.net.UnknownHostException: kafka-cluster-zookeeper-2.kafka-cluster-zookeeper-nodes.kafka-ns.svc: Name or service not known
	at java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method) ~[?:?]
	at java.net.InetAddress$PlatformNameService.lookupAllHostAddr(InetAddress.java:934) ~[?:?]
	at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1543) ~[?:?]
	at java.net.InetAddress$NameServiceAddresses.get(InetAddress.java:852) ~[?:?]
	at java.net.InetAddress.getAllByName0(InetAddress.java:1533) ~[?:?]
	at java.net.InetAddress.getAllByName(InetAddress.java:1385) ~[?:?]
	at java.net.InetAddress.getAllByName(InetAddress.java:1306) ~[?:?]
	at org.apache.zookeeper.client.StaticHostProvider$1.getAllByName(StaticHostProvider.java:88) ~[org.apache.zookeeper.zookeeper-3.8.3.jar:3.8.3]
	at org.apache.zookeeper.client.StaticHostProvider.resolve(StaticHostProvider.java:141) ~[org.apache.zookeeper.zookeeper-3.8.3.jar:3.8.3]
	at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:368) ~[org.apache.zookeeper.zookeeper-3.8.3.jar:3.8.3]
	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1204) ~[org.apache.zookeeper.zookeeper-3.8.3.jar:3.8.3]
2024-05-04 03:23:22 WARN  ClientCnxnSocketNetty:151 - future isn't success.
java.net.UnknownHostException: kafka-cluster-zookeeper-2.kafka-cluster-zookeeper-nodes.kafka-ns.svc
	at java.net.InetAddress$CachedAddresses.get(InetAddress.java:801) ~[?:?]
	at java.net.InetAddress.getAllByName0(InetAddress.java:1533) ~[?:?]
	at java.net.InetAddress.getAllByName(InetAddress.java:1385) ~[?:?]
	at java.net.InetAddress.getAllByName(InetAddress.java:1306) ~[?:?]
	at java.net.InetAddress.getByName(InetAddress.java:1256) ~[?:?]
	at io.netty.util.internal.SocketUtils$8.run(SocketUtils.java:156) ~[io.netty.netty-common-4.1.107.Final.jar:4.1.107.Final]
	at io.netty.util.internal.SocketUtils$8.run(SocketUtils.java:153) ~[io.netty.netty-common-4.1.107.Final.jar:4.1.107.Final]
	at java.security.AccessController.doPrivileged(AccessController.java:569) ~[?:?]
	at io.netty.util.internal.SocketUtils.addressByName(SocketUtils.java:153) ~[io.netty.netty-common-4.1.107.Final.jar:4.1.107.Final]
	at io.netty.resolver.DefaultNameResolver.doResolve(DefaultNameResolver.java:41) ~[io.netty.netty-resolver-4.1.107.Final.jar:4.1.107.Final]
	at io.netty.resolver.SimpleNameResolver.resolve(SimpleNameResolver.java:61) ~[io.netty.netty-resolver-4.1.107.Final.jar:4.1.107.Final]
	at io.netty.resolver.SimpleNameResolver.resolve(SimpleNameResolver.java:53) ~[io.netty.netty-resolver-4.1.107.Final.jar:4.1.107.Final]
	at io.netty.resolver.InetSocketAddressResolver.doResolve(InetSocketAddressResolver.java:55) ~[io.netty.netty-resolver-4.1.107.Final.jar:4.1.107.Final]
	at io.netty.resolver.InetSocketAddressResolver.doResolve(InetSocketAddressResolver.java:31) ~[io.netty.netty-resolver-4.1.107.Final.jar:4.1.107.Final]
	at io.netty.resolver.AbstractAddressResolver.resolve(AbstractAddressResolver.java:106) ~[io.netty.netty-resolver-4.1.107.Final.jar:4.1.107.Final]
	at io.netty.bootstrap.Bootstrap.doResolveAndConnect0(Bootstrap.java:220) ~[io.netty.netty-transport-4.1.107.Final.jar:4.1.107.Final]
	at io.netty.bootstrap.Bootstrap.access$000(Bootstrap.java:47) ~[io.netty.netty-transport-4.1.107.Final.jar:4.1.107.Final]
	at io.netty.bootstrap.Bootstrap$1.operationComplete(Bootstrap.java:189) ~[io.netty.netty-transport-4.1.107.Final.jar:4.1.107.Final]
	at io.netty.bootstrap.Bootstrap$1.operationComplete(Bootstrap.java:175) ~[io.netty.netty-transport-4.1.107.Final.jar:4.1.107.Final]
	at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) ~[io.netty.netty-common-4.1.107.Final.jar:4.1.107.Final]
	at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) ~[io.netty.netty-common-4.1.107.Final.jar:4.1.107.Final]
	at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) ~[io.netty.netty-common-4.1.107.Final.jar:4.1.107.Final]
	at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) ~[io.netty.netty-common-4.1.107.Final.jar:4.1.107.Final]
	at io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625) ~[io.netty.netty-common-4.1.107.Final.jar:4.1.107.Final]
	at io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105) ~[io.netty.netty-common-4.1.107.Final.jar:4.1.107.Final]
	at io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) ~[io.netty.netty-transport-4.1.107.Final.jar:4.1.107.Final]
	at io.netty.channel.AbstractChannel$AbstractUnsafe.safeSetSuccess(AbstractChannel.java:990) ~[io.netty.netty-transport-4.1.107.Final.jar:4.1.107.Final]
	at io.netty.channel.AbstractChannel$AbstractUnsafe.register0(AbstractChannel.java:516) ~[io.netty.netty-transport-4.1.107.Final.jar:4.1.107.Final]
	at io.netty.channel.AbstractChannel$AbstractUnsafe.access$200(AbstractChannel.java:429) ~[io.netty.netty-transport-4.1.107.Final.jar:4.1.107.Final]
	at io.netty.channel.AbstractChannel$AbstractUnsafe$1.run(AbstractChannel.java:486) ~[io.netty.netty-transport-4.1.107.Final.jar:4.1.107.Final]
	at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173) ~[io.netty.netty-common-4.1.107.Final.jar:4.1.107.Final]
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166) ~[io.netty.netty-common-4.1.107.Final.jar:4.1.107.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470) ~[io.netty.netty-common-4.1.107.Final.jar:4.1.107.Final]
	at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:413) ~[io.netty.netty-transport-classes-epoll-4.1.107.Final.jar:4.1.107.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[io.netty.netty-common-4.1.107.Final.jar:4.1.107.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[io.netty.netty-common-4.1.107.Final.jar:4.1.107.Final]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[io.netty.netty-common-4.1.107.Final.jar:4.1.107.Final]
	at java.lang.Thread.run(Thread.java:840) ~[?:?]

It is coming for every instance.

I created another pod with networking utilities in same cluster to verify dig.

nginx-netshoot-84dcf46b67-d2nch:~# dig kafka-cluster-zookeeper-2.kafka-cluster-zookeeper-nodes.kafka-ns.svc

; <<>> DiG 9.18.21 <<>> kafka-cluster-zookeeper-2.kafka-cluster-zookeeper-nodes.kafka-ns.svc
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 65483
;; flags: qr rd ra ad; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
; COOKIE: 275e1b5067d44ed2 (echoed)
;; QUESTION SECTION:
;kafka-cluster-zookeeper-2.kafka-cluster-zookeeper-nodes.kafka-ns.svc. IN A

;; AUTHORITY SECTION:
.			30	IN	SOA	a.root-servers.net. nstld.verisign-grs.com. 2024050303 1800 900 604800 86400

;; Query time: 0 msec
;; SERVER: 10.96.5.5#53(10.96.5.5) (UDP)
;; WHEN: Sat May 04 03:18:33 UTC 2024
;; MSG SIZE  rcvd: 184

It is resolving. So not sure, why it is complaining.

~ $ k get pods -n kafka-ns -o wide
NAME                        READY   STATUS    RESTARTS   AGE   IP             NODE          NOMINATED NODE   READINESS GATES
kafka-cluster-zookeeper-0   1/1     Running   0          12m   172.0.60.147   172.0.1.34    <none>           <none>
kafka-cluster-zookeeper-1   1/1     Running   0          12m   172.0.43.29    172.0.1.8     <none>           <none>
kafka-cluster-zookeeper-2   1/1     Running   0          12m   172.0.51.243   172.0.1.172   <none>           <none>

Logs of zookeeper-0

2024-05-04 03:20:58,986 ERROR Failed to verify hostname: 172.0.51.243 (org.apache.zookeeper.common.ZKTrustManager) [ListenerHandler-/0.0.0.0:3888]
javax.net.ssl.SSLPeerUnverifiedException: Certificate for <172.0.51.243> doesn't match any of the subject alternative names: [kafka-cluster-zookeeper-client.kafka-ns.svc, *.kafka-cluster-zookeeper-client.kafka-ns.svc.cluster.local, *.kafka-cluster-zookeeper-client.kafka-ns.svc, *.kafka-cluster-zookeeper-nodes.kafka-ns.svc, kafka-cluster-zookeeper-client.kafka-ns, kafka-cluster-zookeeper-client, *.kafka-cluster-zookeeper-nodes.kafka-ns.svc.cluster.local, kafka-cluster-zookeeper-2.kafka-cluster-zookeeper-nodes.kafka-ns.svc.cluster.local, kafka-cluster-zookeeper-2, kafka-cluster-zookeeper-2.kafka-cluster-zookeeper-nodes.kafka-ns.svc, kafka-cluster-zookeeper-client.kafka-ns.svc.cluster.local]
	at org.apache.zookeeper.common.ZKHostnameVerifier.matchIPAddress(ZKHostnameVerifier.java:197)
	at org.apache.zookeeper.common.ZKHostnameVerifier.verify(ZKHostnameVerifier.java:165)
	at org.apache.zookeeper.common.ZKTrustManager.performHostVerification(ZKTrustManager.java:189)
	at org.apache.zookeeper.common.ZKTrustManager.checkClientTrusted(ZKTrustManager.java:93)
	at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkClientCerts(CertificateMessage.java:688)
	at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.onCertificate(CertificateMessage.java:411)
	at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.consume(CertificateMessage.java:375)
	at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:396)
	at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:480)
	at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:458)
	at java.base/sun.security.ssl.TransportContext.dispatch(TransportContext.java:201)
	at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:172)
	at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1510)
	at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1425)
	at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:455)
	at java.base/sun.security.ssl.SSLSocketImpl.ensureNegotiated(SSLSocketImpl.java:925)
	at java.base/sun.security.ssl.SSLSocketImpl.getSession(SSLSocketImpl.java:371)
	at org.apache.zookeeper.server.quorum.UnifiedServerSocket$UnifiedSocket.detectMode(UnifiedServerSocket.java:269)
	at org.apache.zookeeper.server.quorum.UnifiedServerSocket$UnifiedSocket.getSocket(UnifiedServerSocket.java:298)
	at org.apache.zookeeper.server.quorum.UnifiedServerSocket$UnifiedSocket.access$400(UnifiedServerSocket.java:172)
	at org.apache.zookeeper.server.quorum.UnifiedServerSocket$UnifiedInputStream.getRealInputStream(UnifiedServerSocket.java:699)
	at org.apache.zookeeper.server.quorum.UnifiedServerSocket$UnifiedInputStream.read(UnifiedServerSocket.java:693)
	at java.base/java.io.BufferedInputStream.fill(BufferedInputStream.java:244)
	at java.base/java.io.BufferedInputStream.read1(BufferedInputStream.java:284)
	at java.base/java.io.BufferedInputStream.read(BufferedInputStream.java:343)
	at java.base/java.io.DataInputStream.readFully(DataInputStream.java:201)
	at java.base/java.io.DataInputStream.readLong(DataInputStream.java:410)
	at org.apache.zookeeper.server.quorum.QuorumCnxManager.handleConnection(QuorumCnxManager.java:602)
	at org.apache.zookeeper.server.quorum.QuorumCnxManager.receiveConnection(QuorumCnxManager.java:555)
	at org.apache.zookeeper.server.quorum.QuorumCnxManager$Listener$ListenerHandler.acceptConnections(QuorumCnxManager.java:1085)
	at org.apache.zookeeper.server.quorum.QuorumCnxManager$Listener$ListenerHandler.run(QuorumCnxManager.java:1039)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:840)
2024-05-04 03:20:58,986 INFO Accepted TLS connection from 172.0.51.243/172.0.51.243:52220 - NONE - SSL_NULL_WITH_NULL_NULL (org.apache.zookeeper.server.quorum.UnifiedServerSocket) [ListenerHandler-/0.0.0.0:3888]
2024-05-04 03:20:58,986 WARN Exception reading or writing challenge (org.apache.zookeeper.server.quorum.QuorumCnxManager) [ListenerHandler-/0.0.0.0:3888]
java.net.SocketException: Socket is closed
	at java.base/sun.security.ssl.SSLSocketImpl.getInputStream(SSLSocketImpl.java:893)
	at org.apache.zookeeper.server.quorum.UnifiedServerSocket$UnifiedInputStream.getRealInputStream(UnifiedServerSocket.java:699)
	at org.apache.zookeeper.server.quorum.UnifiedServerSocket$UnifiedInputStream.read(UnifiedServerSocket.java:693)
	at java.base/java.io.BufferedInputStream.fill(BufferedInputStream.java:244)
	at java.base/java.io.BufferedInputStream.read1(BufferedInputStream.java:284)
	at java.base/java.io.BufferedInputStream.read(BufferedInputStream.java:343)
	at java.base/java.io.DataInputStream.readFully(DataInputStream.java:201)
	at java.base/java.io.DataInputStream.readLong(DataInputStream.java:410)
	at org.apache.zookeeper.server.quorum.QuorumCnxManager.handleConnection(QuorumCnxManager.java:602)
	at org.apache.zookeeper.server.quorum.QuorumCnxManager.receiveConnection(QuorumCnxManager.java:555)
	at org.apache.zookeeper.server.quorum.QuorumCnxManager$Listener$ListenerHandler.acceptConnections(QuorumCnxManager.java:1085)
	at org.apache.zookeeper.server.quorum.QuorumCnxManager$Listener$ListenerHandler.run(QuorumCnxManager.java:1039)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:840)
2024-05-04 03:20:59,698 INFO Processing ruok command from /127.0.0.1:45300 (org.apache.zookeeper.server.NettyServerCnxn) [nioEventLoopGroup-4-15]
2024-05-04 03:20:59,698 INFO Processing ruok command from /127.0.0.1:45316 (org.apache.zookeeper.server.NettyServerCnxn) [nioEventLoopGroup-4-16]
2024-05-04 03:21:09,697 INFO Processing ruok command from /127.0.0.1:60438 (org.apache.zookeeper.server.NettyServerCnxn) [nioEventLoopGroup-4-2]
2024-05-04 03:21:09,697 INFO Processing ruok command from /127.0.0.1:60426 (org.apache.zookeeper.server.NettyServerCnxn) [nioEventLoopGroup-4-1]
2024-05-04 03:21:19,701 INFO Processing ruok command from /127.0.0.1:37120 (org.apache.zookeeper.server.NettyServerCnxn) [nioEventLoopGroup-4-3]
2024-05-04 03:21:19,701 INFO Processing ruok command from /127.0.0.1:37118 (org.apache.zookeeper.server.NettyServerCnxn) [nioEventLoopGroup-4-4]
2024-05-04 03:21:29,699 INFO Processing ruok command from /127.0.0.1:59360 (org.apache.zookeeper.server.NettyServerCnxn) [nioEventLoopGroup-4-5]
2024-05-04 03:21:29,699 INFO Processing ruok command from /127.0.0.1:59376 (org.apache.zookeeper.server.NettyServerCnxn) [nioEventLoopGroup-4-6]
2024-05-04 03:21:32,656 WARN Cannot open channel to 2 at election address kafka-cluster-zookeeper-1.kafka-cluster-zookeeper-nodes.kafka-ns.svc/<unresolved>:3888 (org.apache.zookeeper.server.quorum.QuorumCnxManager) [QuorumConnectionThread-[myid=1]-15]
java.net.UnknownHostException: kafka-cluster-zookeeper-1.kafka-cluster-zookeeper-nodes.kafka-ns.svc
	at java.base/sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:572)
	at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:327)
	at java.base/java.net.Socket.connect(Socket.java:633)
	at java.base/sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:304)
	at org.apache.zookeeper.server.quorum.QuorumCnxManager.initiateConnection(QuorumCnxManager.java:384)
	at org.apache.zookeeper.server.quorum.QuorumCnxManager$QuorumConnectionReqThread.run(QuorumCnxManager.java:458)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:840)
2024-05-04 03:21:39,695 INFO Processing ruok command from /127.0.0.1:35614 (org.apache.zookeeper.server.NettyServerCnxn) [nioEventLoopGroup-4-7]
2024-05-04 03:21:39,697 INFO Processing ruok command from /127.0.0.1:35626 (org.apache.zookeeper.server.NettyServerCnxn) [nioEventLoopGroup-4-8]
2024-05-04 03:21:48,967 INFO Received connection request from /172.0.51.243:46118 (org.apache.zookeeper.server.quorum.QuorumCnxManager) [ListenerHandler-/0.0.0.0:3888]
2024-05-04 03:21:49,701 INFO Processing ruok command from /127.0.0.1:55850 (org.apache.zookeeper.server.NettyServerCnxn) [nioEventLoopGroup-4-10]
2024-05-04 03:21:49,701 INFO Processing ruok command from /127.0.0.1:55846 (org.apache.zookeeper.server.NettyServerCnxn) [nioEventLoopGroup-4-9]
2024-05-04 03:21:52,676 INFO Notification time out: 51200 ms (org.apache.zookeeper.server.quorum.FastLeaderElection) [QuorumPeer[myid=1](plain=127.0.0.1:12181)(secure=0.0.0.0:2181)]
2024-05-04 03:21:52,676 WARN Cannot open channel to 3 at election address kafka-cluster-zookeeper-2.kafka-cluster-zookeeper-nodes.kafka-ns.svc/<unresolved>:3888 (org.apache.zookeeper.server.quorum.QuorumCnxManager) [QuorumConnectionThread-[myid=1]-16]
java.net.UnknownHostException: kafka-cluster-zookeeper-2.kafka-cluster-zookeeper-nodes.kafka-ns.svc
	at java.base/sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:572)
	at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:327)
	at java.base/java.net.Socket.connect(Socket.java:633)
	at java.base/sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:304)
	at org.apache.zookeeper.server.quorum.QuorumCnxManager.initiateConnection(QuorumCnxManager.java:384)
	at org.apache.zookeeper.server.quorum.QuorumCnxManager$QuorumConnectionReqThread.run(QuorumCnxManager.java:458)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:840)
2024-05-04 03:21:58,988 ERROR Failed to verify host address: 172.0.51.243 (org.apache.zookeeper.common.ZKTrustManager) [ListenerHandler-/0.0.0.0:3888]
javax.net.ssl.SSLPeerUnverifiedException: Certificate for <172.0.51.243> doesn't match any of the subject alternative names: [kafka-cluster-zookeeper-client.kafka-ns.svc, *.kafka-cluster-zookeeper-client.kafka-ns.svc.cluster.local, *.kafka-cluster-zookeeper-client.kafka-ns.svc, *.kafka-cluster-zookeeper-nodes.kafka-ns.svc, kafka-cluster-zookeeper-client.kafka-ns, kafka-cluster-zookeeper-client, *.kafka-cluster-zookeeper-nodes.kafka-ns.svc.cluster.local, kafka-cluster-zookeeper-2.kafka-cluster-zookeeper-nodes.kafka-ns.svc.cluster.local, kafka-cluster-zookeeper-2, kafka-cluster-zookeeper-2.kafka-cluster-zookeeper-nodes.kafka-ns.svc, kafka-cluster-zookeeper-client.kafka-ns.svc.cluster.local]
	at org.apache.zookeeper.common.ZKHostnameVerifier.matchIPAddress(ZKHostnameVerifier.java:197)
	at org.apache.zookeeper.common.ZKHostnameVerifier.verify(ZKHostnameVerifier.java:165)
	at org.apache.zookeeper.common.ZKTrustManager.performHostVerification(ZKTrustManager.java:180)
	at org.apache.zookeeper.common.ZKTrustManager.checkClientTrusted(ZKTrustManager.java:93)
	at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkClientCerts(CertificateMessage.java:688)
	at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.onCertificate(CertificateMessage.java:411)
	at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.consume(CertificateMessage.java:375)
	at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:396)
	at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:480)
	at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:458)
	at java.base/sun.security.ssl.TransportContext.dispatch(TransportContext.java:201)
	at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:172)
	at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1510)
	at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1425)
	at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:455)
	at java.base/sun.security.ssl.SSLSocketImpl.ensureNegotiated(SSLSocketImpl.java:925)
	at java.base/sun.security.ssl.SSLSocketImpl.getSession(SSLSocketImpl.java:371)
	at org.apache.zookeeper.server.quorum.UnifiedServerSocket$UnifiedSocket.detectMode(UnifiedServerSocket.java:269)
	at org.apache.zookeeper.server.quorum.UnifiedServerSocket$UnifiedSocket.getSocket(UnifiedServerSocket.java:298)
	at org.apache.zookeeper.server.quorum.UnifiedServerSocket$UnifiedSocket.access$400(UnifiedServerSocket.java:172)
	at org.apache.zookeeper.server.quorum.UnifiedServerSocket$UnifiedInputStream.getRealInputStream(UnifiedServerSocket.java:699)
	at org.apache.zookeeper.server.quorum.UnifiedServerSocket$UnifiedInputStream.read(UnifiedServerSocket.java:693)
	at java.base/java.io.BufferedInputStream.fill(BufferedInputStream.java:244)
	at java.base/java.io.BufferedInputStream.read1(BufferedInputStream.java:284)
	at java.base/java.io.BufferedInputStream.read(BufferedInputStream.java:343)
	at java.base/java.io.DataInputStream.readFully(DataInputStream.java:201)
	at java.base/java.io.DataInputStream.readLong(DataInputStream.java:410)
	at org.apache.zookeeper.server.quorum.QuorumCnxManager.handleConnection(QuorumCnxManager.java:602)
	at org.apache.zookeeper.server.quorum.QuorumCnxManager.receiveConnection(QuorumCnxManager.java:555)
	at org.apache.zookeeper.server.quorum.QuorumCnxManager$Listener$ListenerHandler.acceptConnections(QuorumCnxManager.java:1085)
	at org.apache.zookeeper.server.quorum.QuorumCnxManager$Listener$ListenerHandler.run(QuorumCnxManager.java:1039)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:840)
2024-05-04 03:21:58,988 ERROR Failed to verify hostname: 172.0.51.243 (org.apache.zookeeper.common.ZKTrustManager) [ListenerHandler-/0.0.0.0:3888]
javax.net.ssl.SSLPeerUnverifiedException: Certificate for <172.0.51.243> doesn't match any of the subject alternative names: [kafka-cluster-zookeeper-client.kafka-ns.svc, *.kafka-cluster-zookeeper-client.kafka-ns.svc.cluster.local, *.kafka-cluster-zookeeper-client.kafka-ns.svc, *.kafka-cluster-zookeeper-nodes.kafka-ns.svc, kafka-cluster-zookeeper-client.kafka-ns, kafka-cluster-zookeeper-client, *.kafka-cluster-zookeeper-nodes.kafka-ns.svc.cluster.local, kafka-cluster-zookeeper-2.kafka-cluster-zookeeper-nodes.kafka-ns.svc.cluster.local, kafka-cluster-zookeeper-2, kafka-cluster-zookeeper-2.kafka-cluster-zookeeper-nodes.kafka-ns.svc, kafka-cluster-zookeeper-client.kafka-ns.svc.cluster.local]
	at org.apache.zookeeper.common.ZKHostnameVerifier.matchIPAddress(ZKHostnameVerifier.java:197)
	at org.apache.zookeeper.common.ZKHostnameVerifier.verify(ZKHostnameVerifier.java:165)
	at org.apache.zookeeper.common.ZKTrustManager.performHostVerification(ZKTrustManager.java:189)
	at org.apache.zookeeper.common.ZKTrustManager.checkClientTrusted(ZKTrustManager.java:93)
	at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkClientCerts(CertificateMessage.java:688)
	at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.onCertificate(CertificateMessage.java:411)
	at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.consume(CertificateMessage.java:375)
	at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:396)
	at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:480)
	at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:458)
	at java.base/sun.security.ssl.TransportContext.dispatch(TransportContext.java:201)
	at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:172)
	at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1510)
	at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1425)
	at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:455)
	at java.base/sun.security.ssl.SSLSocketImpl.ensureNegotiated(SSLSocketImpl.java:925)
	at java.base/sun.security.ssl.SSLSocketImpl.getSession(SSLSocketImpl.java:371)
	at org.apache.zookeeper.server.quorum.UnifiedServerSocket$UnifiedSocket.detectMode(UnifiedServerSocket.java:269)
	at org.apache.zookeeper.server.quorum.UnifiedServerSocket$UnifiedSocket.getSocket(UnifiedServerSocket.java:298)
	at org.apache.zookeeper.server.quorum.UnifiedServerSocket$UnifiedSocket.access$400(UnifiedServerSocket.java:172)
	at org.apache.zookeeper.server.quorum.UnifiedServerSocket$UnifiedInputStream.getRealInputStream(UnifiedServerSocket.java:699)
	at org.apache.zookeeper.server.quorum.UnifiedServerSocket$UnifiedInputStream.read(UnifiedServerSocket.java:693)
	at java.base/java.io.BufferedInputStream.fill(BufferedInputStream.java:244)
	at java.base/java.io.BufferedInputStream.read1(BufferedInputStream.java:284)
	at java.base/java.io.BufferedInputStream.read(BufferedInputStream.java:343)
	at java.base/java.io.DataInputStream.readFully(DataInputStream.java:201)
	at java.base/java.io.DataInputStream.readLong(DataInputStream.java:410)
	at org.apache.zookeeper.server.quorum.QuorumCnxManager.handleConnection(QuorumCnxManager.java:602)
	at org.apache.zookeeper.server.quorum.QuorumCnxManager.receiveConnection(QuorumCnxManager.java:555)
	at org.apache.zookeeper.server.quorum.QuorumCnxManager$Listener$ListenerHandler.acceptConnections(QuorumCnxManager.java:1085)
	at org.apache.zookeeper.server.quorum.QuorumCnxManager$Listener$ListenerHandler.run(QuorumCnxManager.java:1039)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:840)
2024-05-04 03:21:58,988 INFO Accepted TLS connection from 172.0.51.243/172.0.51.243:46118 - NONE - SSL_NULL_WITH_NULL_NULL (org.apache.zookeeper.server.quorum.UnifiedServerSocket) [ListenerHandler-/0.0.0.0:3888]
2024-05-04 03:21:58,988 WARN Exception reading or writing challenge (org.apache.zookeeper.server.quorum.QuorumCnxManager) [ListenerHandler-/0.0.0.0:3888]
java.net.SocketException: Socket is closed
	at java.base/sun.security.ssl.SSLSocketImpl.getInputStream(SSLSocketImpl.java:893)
	at org.apache.zookeeper.server.quorum.UnifiedServerSocket$UnifiedInputStream.getRealInputStream(UnifiedServerSocket.java:699)
	at org.apache.zookeeper.server.quorum.UnifiedServerSocket$UnifiedInputStream.read(UnifiedServerSocket.java:693)
	at java.base/java.io.BufferedInputStream.fill(BufferedInputStream.java:244)
	at java.base/java.io.BufferedInputStream.read1(BufferedInputStream.java:284)
	at java.base/java.io.BufferedInputStream.read(BufferedInputStream.java:343)
	at java.base/java.io.DataInputStream.readFully(DataInputStream.java:201)
	at java.base/java.io.DataInputStream.readLong(DataInputStream.java:410)
	at org.apache.zookeeper.server.quorum.QuorumCnxManager.handleConnection(QuorumCnxManager.java:602)
	at org.apache.zookeeper.server.quorum.QuorumCnxManager.receiveConnection(QuorumCnxManager.java:555)
	at org.apache.zookeeper.server.quorum.QuorumCnxManager$Listener$ListenerHandler.acceptConnections(QuorumCnxManager.java:1085)
	at org.apache.zookeeper.server.quorum.QuorumCnxManager$Listener$ListenerHandler.run(QuorumCnxManager.java:1039)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:840)

[scholzj]

Well, you seem to be still getting UnknownHostException, so it does not look like the DNS issue is completely resolved maybe? The TLS hostname verification might be also related to DNS and how it does reverse lookup. But from the log it is not clear if it succeeded or failed eventually.