[Bug] kafka cluster pods don't start

[juangon]

Describe the bug
When using strimzi operator to try to start a Kafka infrastructure, kafka cluster pods don't start.

To Reproduce
Steps to reproduce the behavior:

1. Create a separate namespace:

kubectl create namespace kafka    

2. Download strimzi YAML and create resources:

curl -L https://github.com/strimzi/strimzi-kafka-operator/releases/download/0.16.0/strimzi-cluster-operator-0.16.0.yaml \
 | sed 's/namespace: .*/namespace: kafka/' \
 | kubectl apply -f - -n kafka

3.kubectl applythis file:

apiVersion: kafka.strimzi.io/v1beta1
kind: Kafka
metadata:
  name: my-cluster
spec:
  kafka:
    version: 2.4.0
    replicas: 3
    listeners:
      plain: {}
      tls: {}
      external:
        type: nodeport
        tls: false
    config:
      offsets.topic.replication.factor: 3
      transaction.state.log.replication.factor: 3
      transaction.state.log.min.isr: 2
      log.message.format.version: '2.3'
    storage:
      type: ephemeral
  zookeeper:
    replicas: 3
    storage:
      type: ephemeral
  entityOperator:
    topicOperator: {}
    userOperator: {}

3. my-cluster-kafka-* Pods doesn't start and keep restarting

Expected behavior
Kafka cluster pods starts properly.

Environment:

• Strimzi version: 0.16.0
• Installation method: YAML files
• Kubernetes cluster: Kubernetes 1.17.0
• Infrastructure: Minikube

YAML files and logs
Attached logs for my-cluster-kafka-0,my-cluster-kafka-1, zookeper and tls-sidecar from one zookeper Pod. (there are 3 pods)
logs-from-kafka-in-my-cluster-kafka-0.txt
logs-from-kafka-in-my-cluster-kafka-1.txt

logs-from-zookeeper-in-my-cluster-zookeeper-0.txt
logs-from-strimzi-cluster-operator-in-strimzi-cluster-operator-6f47f586f7-rlv96.txt
logs-from-tls-sidecar-in-my-cluster-zookeeper-1.txt

[juangon]

BTW just updated the script to use 0.16.1, but still failing...

[ppatierno]

can you provide the cluster operator log please? What are doing the ZK pods? it seems that connection with ZK is lost at some point. Are the TLS sidecars up and running?

[juangon]

Thanks for answering @ppatierno . I am collecting cluster operator log now, but how I know if the TLS sidecars are up and running?

[juangon]

@ppatierno I added logs from one of zookeper Pods and strimzi cluster operator.

[ppatierno]

I don't think ZK pods are working fine actually, so the problem with the Kafka ones. On each ZK pod there is the ZK container and the tls-sidecar container. Can you take the log of the tls-sidecar container for each of ZK pod please?

[juangon]

I don't think ZK pods are working fine actually, so the problem with the Kafka ones. On each ZK pod there is the ZK container and the tls-sidecar container. Can you take the log of the tls-sidecar container for each of ZK pod please?

I see from kubernetes dashboard there is a tls-sidecar section within zookeper-XX pod, but don't see any option to get logs. Can you guide me for doing so? Thanks!

[juangon]

Nevermind, I found it @ppatierno . I just attached it

[juangon]

Hi @ppatierno . Can I help sending more data to check?
Thanks!

[scholzj]

I will close this since there was no update for more than 10 days. If you have something more feel free to open a new issue, reopen this one or get in touch with us on Slack or mailing list.