KafkaConnect Connector Resource not able to connect to kafka cluster #5442

alpacino12 · 2021-08-17T16:58:32Z

alpacino12
Aug 17, 2021

I have deployed a 3 node Strimzi kafka Cluster having strimzi operator version 0.23 inside Azure Kubernetes Cluster .The Cluster and kafka/zookeeper pods are up and running . Strimzi kafka image used is strimzi/kafka:0.23.0-kafka-2.7.0
tushar@Azure:~$ kubectl get pods -n strimzi-kafka-aks-cluster
NAME READY STATUS RESTARTS AGE
my-connect-cluster-connect-7b4cd65946-c9qjg 0/1 CrashLoopBackOff 25 112m
strimzi-kafka-aks-cluster-entity-operator-665645cc-wcgdn 3/3 Running 0 11d
strimzi-kafka-aks-cluster-kafka-0 1/1 Running 0 11d
strimzi-kafka-aks-cluster-kafka-1 1/1 Running 0 11d
strimzi-kafka-aks-cluster-kafka-2 1/1 Running 0 11d
strimzi-kafka-aks-cluster-zookeeper-0 1/1 Running 0 11d
strimzi-kafka-aks-cluster-zookeeper-1 1/1 Running 0 11d
strimzi-kafka-aks-cluster-zookeeper-2 1/1 Running 0 11d

I'm trying to create a kafkaconnect cluster but the connect container keeps on crashing . Here is the yaml deployment i'm using

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnect
metadata:
  name: my-connect-cluster
#  annotations:
#  # use-connector-resources configures this KafkaConnect
#  # to use KafkaConnector resources to avoid
#  # needing to call the Connect REST API directly
#    strimzi.io/use-connector-resources: "true"
spec:
  replicas: 1
  bootstrapServers: strimzi-kafka-aks-cluster-kafka-extnl-bootstrap:9094
  #tls:
   # trustedCertificates:
    # - secretName: strimzi-kafka-aks-cluster-cluster-ca-cert
     #   certificate: ca.crt
  config:
    group.id: connect-cluster
    offset.storage.topic: connect-cluster-offsets
    config.storage.topic: connect-cluster-configs
    status.storage.topic: connect-cluster-status
    # -1 means it will use the default replication factor configured in the broker
    config.storage.replication.factor: -1
    offset.storage.replication.factor: -1

I have tried with both tls config map and without , with spec.image tag and without but each time it gives connection failed to kafka broker error

When i describe the failed pod it says

Liveness probe failed: Get "http://10.244.5.25:8083/": dial tcp 10.244.5.25:8083: connect:connection refused

logs :

2021-08-17 16:43:58,000 ERROR Stopping due to error (org.apache.kafka.connect.cli.ConnectDistributed) [main]
org.apache.kafka.connect.errors.ConnectException: Failed to connect to and describe Kafka cluster. Check worker's broker connection and securityproperties.
at org.apache.kafka.connect.util.ConnectUtils.lookupKafkaClusterId(ConnectUtils.java:70)
at org.apache.kafka.connect.util.ConnectUtils.lookupKafkaClusterId(ConnectUtils.java:51)
at org.apache.kafka.connect.cli.ConnectDistributed.startConnect(ConnectDistributed.java:97)
at org.apache.kafka.connect.cli.ConnectDistributed.main(ConnectDistributed.java:80)
Caused by: java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Call(callName=listNodes, deadlineMs=1629218637936, tries=1, nextAllowedTryMs=1629218638070) timed out at 1629218637970 after 1 attempt(s)
at org.apache.kafka.common.internals.KafkaFutureImpl.wrapAndThrow(KafkaFutureImpl.java:45)
at org.apache.kafka.common.internals.KafkaFutureImpl.access$000(KafkaFutureImpl.java:32)
at org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:89)
at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:260)
at org.apache.kafka.connect.util.ConnectUtils.lookupKafkaClusterId(ConnectUtils.java:64)
... 3 more
Caused by: org.apache.kafka.common.errors.TimeoutException: Call(callName=listNodes, deadlineMs=1629218637936, tries=1, nextAllowedTryMs=1629218638070) timed out at 1629218637970 after 1 attempt(s)
Caused by: org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment. Call: listNodes

Is there any issue with the deployement here ? Why is the connect instance not able to connect to the kafka cluster

scholzj · 2021-08-17T18:25:49Z

scholzj
Aug 17, 2021
Maintainer

If you format the YAML as a code, it will be much more readable and easier to detect any whitespaces issues etc. Also, can you please share the Kafka custom resource with your Kafka cluster configuration? Without that it is hard to say if you configured it correctly or not.

0 replies

alpacino12 · 2021-08-18T11:54:34Z

alpacino12
Aug 18, 2021
Author

Here's the yaml file i'm using to start the kafkaConnect pod

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnect
metadata:
  name: my-connect-cluster
#  annotations:
#  # use-connector-resources configures this KafkaConnect
#  # to use KafkaConnector resources to avoid
#  # needing to call the Connect REST API directly
#    strimzi.io/use-connector-resources: "true"
spec:
  version: 2.7.0
  replicas: 1
  bootstrapServers: kafka-gsmdp-cluster-kafka-bootstrap:9093
  tls:
    trustedCertificates:
      - secretName:  kafka-gsmdp-cluster-cluster-ca-cert
        certificate: ca.crt
  config:
    group.id: connect-cluster
    offset.storage.topic: connect-cluster-offsets
    config.storage.topic: connect-cluster-configs
    status.storage.topic: connect-cluster-status
    # -1 means it will use the default replication factor configured in the broker
    config.storage.replication.factor: -1
    offset.storage.replication.factor: -1
    status.storage.replication.factor: -1
```

I created the cluster via strimzi helm chart , updated values.yaml file like below

`# Default values for strimzi-kafka.
# This is a YAML-formatted file.
# Declare variables to be passed into your templates.
# provide namespace for strimzi kafka cluster
namespace: strimzi-operator-aks-cluster

#Provide your kafka cluster name
cluster:
  kafkaname: kafka-gsmdp-cluster
#Provide number of kafka brokers need in kafka cluster
replicaCount: 3

# Mention internal and external ports for connecting to Kafka cluster
listeners:
  internalport: 9093
  extrnlport: 9094
# provide kafka resources
kafkaresources:
  storageclass: default
  volumecapacity: 10Gi
  reqmemory: 500m
  reqcpu: 500m
  limitmemory: 4Gi
  limitcpu: 500m

# Provide Zookeeper resources according requirements
zkresources:
  volumecapacity: 10Gi
  reqmemory: 500m
  reqcpu: 250m
  limitmemory: 2Gi
  limitcpu: 500m
# Provide versions according to  kafka version using in cluster
versions:
  logversion: 2.7
  interbrokerversion: 2.7

# Provide user details for authentication while connecting to cluster
strimziuser:
  kafkauser: my-user
  authtype: scram-sha-512

# Run the below command in your terminal and update the value accordingly in below section
# echo -n '{"auths":{"gateway-sesk1.nsc.internal.ericsson.com:30048":{"username":"etxxxxx","password":"Dv1xxxxxxxxxxaRU"}}}' |base64  | tr -d \\n
harborsecret: eyJhdXRocyI6eyJnYXRld2F5LXNlc2sxLm5zYy5pbnRlcm5hbC5lcmljc3Nvbi5jb206MzAwNDgiOnsidXNlcm5hbWUiOiJldGhvbmFnIiwicGFzc3dvcmQiOiJEdjFSZmNTMlhTTmd0UDRJdXJxOVdESkRhc2FtNWFSVSJ9fX0=

6 replies

scholzj Aug 18, 2021
Maintainer

There is no Strimzi Helm Chart for creating the Kafka resource. We have only a Helm Chart for the operator it self. So it is probably some custom thing and I do not know how the cluster is configured from the Helm Chart values. You would need to provide the actual Kafka resource. You should be able to do kubectl get kafka -o yaml and share the output.

alpacino12 Aug 18, 2021
Author

apiVersion: v1
items:
- apiVersion: kafka.strimzi.io/v1beta2
  kind: Kafka
  metadata:
    annotations:
      meta.helm.sh/release-name: gsmdp-aks
      meta.helm.sh/release-namespace: strimzi-operator-aks-cluster
    creationTimestamp: "2021-08-18T07:03:32Z"
    generation: 1
    labels:
      app.kubernetes.io/managed-by: Helm
    name: kafka-gsmdp-cluster
    namespace: strimzi-operator-aks-cluster
    resourceVersion: "2380022"
    uid: dde3136c-52b8-47ce-a585-411b708af724
  spec:
    entityOperator:
      topicOperator: {}
      userOperator: {}
    kafka:
      config:
        inter.broker.protocol.version: 2.7
        log.message.format.version: 2.7
        offsets.topic.replication.factor: 3
        transaction.state.log.min.isr: 3
        transaction.state.log.replication.factor: 3
      listeners:
      - authentication:
          type: scram-sha-512
        name: tls
        port: 9093
        tls: true
        type: internal
      - authentication:
          type: scram-sha-512
        name: extnl
        port: 9094
        tls: true
        type: loadbalancer
      replicas: 3
      resources:
        limits:
          cpu: 500m
          memory: 4Gi
        requests:
          cpu: 500m
          memory: 500m
      storage:
        class: default
        deleteClaim: false
        size: 10Gi
        type: persistent-claim
      template:
        pod:
          imagePullSecrets:
          - name: regcred-kafka
      version: 2.7.0
    zookeeper:
      replicas: 3
      resources:
        limits:
          cpu: 500m
          memory: 2Gi
        requests:
          cpu: 250m
          memory: 500m
      storage:
        class: default
        deleteClaim: false
        size: 10Gi
        type: persistent-claim
      template:
        pod:
          imagePullSecrets:
          - name: regcred-kafka
  status:
    clusterId: 2PBMKszbT7Cw2XXi5SMX3w
    conditions:
    - lastTransitionTime: "2021-08-18T07:07:13.356Z"
      status: "True"
      type: Ready
    listeners:
    - addresses:
      - host: kafka-gsmdp-cluster-kafka-bootstrap.strimzi-operator-aks-cluster.svc
        port: 9093
      bootstrapServers: kafka-gsmdp-cluster-kafka-bootstrap.strimzi-operator-aks-cluster.svc:9093
      certificates:
      - |
        -----BEGIN CERTIFICATE-----
        MIIDLTCCAhWgAwIBAgIJAKcB84Wh2gWMMA0GCSqGSIb3DQEBCwUAMC0xEzARBgNV
        BAoMCmlvLnN0cmltemkxFjAUBgNVBAMMDWNsdXN0ZXItY2EgdjAwHhcNMjEwODE4
        MDcwMzMzWhcNMjIwODE4MDcwMzMzWjAtMRMwEQYDVQQKDAppby5zdHJpbXppMRYw
        FAYDVQQDDA1jbHVzdGVyLWNhIHYwMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIB
        CgKCAQEA6Zbd7RDI04V5lqXe/B+IE/hiXW/CCL1LpjUcfm+NwQDQhYfff3hjZ0EU
        YC+/03f6pnm3jXfLNdxxMeMoO908DtjAIaF053L2W8FMEFkrRzMK+ZMY1FUZ4qFD
        YFPQOgri4B+sar10LhzEQdenuDVws1bidtIppz+2yesqvoJsV4jWz26qh9Smb5m+
        uPj29nJmEF44v3cGbHM4UMs1F7rYINFOjtrSiShKlzDwYClTMaKKz+TPGy2ye4JD
        PqjPXymYw/x+RKSKHl9w4QrzYv5qnGnMmCp3tE2al4yTKvmLFxpxNzA66+4PKcxz
        lipmar1hZhzLzoo7Zi2gTGDjPDvQjQIDAQABo1AwTjAdBgNVHQ4EFgQUFARyZA2v
        VkztrLnTJ2R10QbmmGcwHwYDVR0jBBgwFoAUFARyZA2vVkztrLnTJ2R10QbmmGcw
        DAYDVR0TBAUwAwEB/zANBgkqhkiG9w0BAQsFAAOCAQEAb/SZAgqxI42MzUxhod1L
        Xp7ABSkJ84QQYFTfK7kvapmAPo8FrL9rRWECwY03iMfvtv+wPqoVOk+A01MhFlHK
        UeNZS0GE19FJjtThQQKsm4j6Cb0sOVHW0/s/liw5UgpTebK+F37YihigDZSvocZ9
        TBn9RtxpDKp6gyWFawfHTraKHfNTSkzYwSqqGqFcPN9t556q3UwLxbeIbw7pFXKx
        ejMmueWQs3nIZdrG7yEBfZkQ/5Kjf8v/o6lwFMChoqMOT2X7TQrvD2xvBykKdSUn
        +zWWQczQ8VdMYHkgkv4Qr/nc2521rzqxSOktwdy0cl/agxRSV5rrMxKcais1hYPo
        2A==
        -----END CERTIFICATE-----
      type: tls
    - addresses:
      - host: 52.182.216.185
        port: 9094
      bootstrapServers: 52.182.216.185:9094
      certificates:
      - |
        -----BEGIN CERTIFICATE-----
        MIIDLTCCAhWgAwIBAgIJAKcB84Wh2gWMMA0GCSqGSIb3DQEBCwUAMC0xEzARBgNV
        BAoMCmlvLnN0cmltemkxFjAUBgNVBAMMDWNsdXN0ZXItY2EgdjAwHhcNMjEwODE4
        MDcwMzMzWhcNMjIwODE4MDcwMzMzWjAtMRMwEQYDVQQKDAppby5zdHJpbXppMRYw
        FAYDVQQDDA1jbHVzdGVyLWNhIHYwMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIB
        CgKCAQEA6Zbd7RDI04V5lqXe/B+IE/hiXW/CCL1LpjUcfm+NwQDQhYfff3hjZ0EU
        YC+/03f6pnm3jXfLNdxxMeMoO908DtjAIaF053L2W8FMEFkrRzMK+ZMY1FUZ4qFD
        YFPQOgri4B+sar10LhzEQdenuDVws1bidtIppz+2yesqvoJsV4jWz26qh9Smb5m+
        uPj29nJmEF44v3cGbHM4UMs1F7rYINFOjtrSiShKlzDwYClTMaKKz+TPGy2ye4JD
        PqjPXymYw/x+RKSKHl9w4QrzYv5qnGnMmCp3tE2al4yTKvmLFxpxNzA66+4PKcxz
        lipmar1hZhzLzoo7Zi2gTGDjPDvQjQIDAQABo1AwTjAdBgNVHQ4EFgQUFARyZA2v
        VkztrLnTJ2R10QbmmGcwHwYDVR0jBBgwFoAUFARyZA2vVkztrLnTJ2R10QbmmGcw
        DAYDVR0TBAUwAwEB/zANBgkqhkiG9w0BAQsFAAOCAQEAb/SZAgqxI42MzUxhod1L
        Xp7ABSkJ84QQYFTfK7kvapmAPo8FrL9rRWECwY03iMfvtv+wPqoVOk+A01MhFlHK
        UeNZS0GE19FJjtThQQKsm4j6Cb0sOVHW0/s/liw5UgpTebK+F37YihigDZSvocZ9
        TBn9RtxpDKp6gyWFawfHTraKHfNTSkzYwSqqGqFcPN9t556q3UwLxbeIbw7pFXKx
        ejMmueWQs3nIZdrG7yEBfZkQ/5Kjf8v/o6lwFMChoqMOT2X7TQrvD2xvBykKdSUn
        +zWWQczQ8VdMYHkgkv4Qr/nc2521rzqxSOktwdy0cl/agxRSV5rrMxKcais1hYPo
        2A==
        -----END CERTIFICATE-----
      type: extnl
    observedGeneration: 1
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

scholzj Aug 18, 2021
Maintainer

Right. So you seem to have SCRAM-SHA-512 authentication on the listener on port 9093. But your connect does not configure that. You will need to add the authentication section there as well. Check the examples here:

strimzi-kafka-operator/examples/security/scram-sha-512-auth/connect.yaml

Line 106 in 8815c8a

authentication:

alpacino12 Aug 18, 2021
Author

As you suggested , after adding the authentication section , the pod is running now . I must admit i'm still quite new to kubernetes/strimzi but your extraordinary insight did it for me . Thanks a lot again!

scholzj Aug 18, 2021
Maintainer

Glad it works now! 👍

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Strimzi

KafkaConnect Connector Resource not able to connect to kafka cluster #5442

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 2 comments 6 replies

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

Select a reply

Strimzi

KafkaConnect Connector Resource not able to connect to kafka cluster #5442

alpacino12 Aug 17, 2021

Replies: 2 comments · 6 replies

scholzj Aug 17, 2021 Maintainer

alpacino12 Aug 18, 2021 Author

scholzj Aug 18, 2021 Maintainer

alpacino12 Aug 18, 2021 Author

scholzj Aug 18, 2021 Maintainer

alpacino12 Aug 18, 2021 Author

scholzj Aug 18, 2021 Maintainer

alpacino12
Aug 17, 2021

Replies: 2 comments 6 replies

scholzj
Aug 17, 2021
Maintainer

alpacino12
Aug 18, 2021
Author

scholzj Aug 18, 2021
Maintainer

alpacino12 Aug 18, 2021
Author

scholzj Aug 18, 2021
Maintainer

alpacino12 Aug 18, 2021
Author

scholzj Aug 18, 2021
Maintainer