Any documentation for the Kafka upgrade across versions

[vperi1730]

Hi Scholzj,

Just a question, do we have any Strimzi documentation which talks about upgrade steps/guidelines from one version to the other.

I am looking for something from 0.17 to any other above versions.

Need some info around this.

[tomncooper]

The docs have sections about upgrading Strimzi. Upgrades usually have to move from one version to the next eg 0.17 -> 0.18 -> 0.19. Jumping versions is not recommended, however it can be done if both versions share support for the same Kafka versions. Again though jumping versions is not recommended.

[tomncooper]

The kafka versions supported by different strimzi versions are listed on the downloads page.

[vperi1730]

Ok...

@tomncooper - This morning I started an upgrade from 17 to 19 by following the strmizi recommended steps below.

7.1. Strimzi and Kafka upgrades
Upgrading Strimzi is a two-stage process. To upgrade brokers and clients without downtime, you must complete the upgrade procedures in the following order:
Update your Cluster Operator to the latest Strimzi version.
Upgrading the Cluster Operator
Upgrade all Kafka brokers and client applications to the latest Kafka version.
Upgrading Kafka

Here are the steps I have followed.

1. Deployed 17 operators.
2. Deployed Kafka with 2.4.0
3. Deployed 19 operator.
4. Updated my Kafka CR to 2.5.0 along with the log version.

After step 4 I started noticing that Kafka pods are going into crashloopbackoff, I was expecting them to get re-created as it should have only a single container without TLS. Following are the state of the pods and logs.

pod/my-cluster-kafka-0                           1/2     CrashLoopBackOff   12         38m
pod/my-cluster-kafka-1                           1/2     CrashLoopBackOff   3          89s
pod/my-cluster-kafka-2                           2/2     Running            0          13h
pod/my-cluster-kafka-exporter-5f9f6869d4-nrblb   1/1     Running            0          13h
pod/my-cluster-zookeeper-0                       1/1     Running            0          42m
pod/my-cluster-zookeeper-1                       1/1     Running            0          40m
pod/my-cluster-zookeeper-2                       1/1     Running            0          41m
pod/strimzi-operator-755764999-2b5dp             1/1     Running            0          47m

[root@lvndev012336 cluster-operator]# kubectl logs -f my-cluster-kafka-1 -n kup20 kafka
++ hostname
++ awk -F- '{print $NF}'
+ STRIMZI_BROKER_ID=1
+ export STRIMZI_BROKER_ID
+ echo STRIMZI_BROKER_ID=1
+ export GC_LOG_ENABLED=false
+ GC_LOG_ENABLED=false
STRIMZI_BROKER_ID=1
+ '[' -z '' ']'
+ export KAFKA_LOG4J_OPTS=-Dlog4j.configuration=file:/opt/kafka/custom-config/log4j.properties
+ KAFKA_LOG4J_OPTS=-Dlog4j.configuration=file:/opt/kafka/custom-config/log4j.properties
+ rm -f /var/opt/kafka/kafka-ready /var/opt/kafka/zk-connected
++ ls /opt/kafka/libs/kafka-agent.jar
+ KAFKA_OPTS=' -javaagent:/opt/kafka/libs/kafka-agent.jar=/var/opt/kafka/kafka-ready:/var/opt/kafka/zk-connected'
+ export KAFKA_OPTS
+ '[' true = true ']'
++ hostname -i
+ KAFKA_JMX_OPTS='-Dcom.sun.management.jmxremote.port=9999 -Dcom.sun.management.jmxremote.rmi.port=9999 -Dcom.sun.management.jmxremote=true -Djava.rmi.server.hostname=240.0.244.183 -Djava.net.preferIPv4Stack=true'
+ '[' -n '' ']'
+ KAFKA_JMX_OPTS='-Dcom.sun.management.jmxremote.port=9999 -Dcom.sun.management.jmxremote.rmi.port=9999 -Dcom.sun.management.jmxremote=true -Djava.rmi.server.hostname=240.0.244.183 -Djava.net.preferIPv4Stack=true -Dcom.sun.management.jmxremote.authenticate=false'
+ '[' -n '' ']'
+ KAFKA_OPTS=' -javaagent:/opt/kafka/libs/kafka-agent.jar=/var/opt/kafka/kafka-ready:/var/opt/kafka/zk-connected -Dcom.sun.management.jmxremote.port=9999 -Dcom.sun.management.jmxremote.rmi.port=9999 -Dcom.sun.management.jmxremote=true -Djava.rmi.server.hostname=240.0.244.183 -Djava.net.preferIPv4Stack=true -Dcom.sun.management.jmxremote.authenticate=false'
+ '[' true = true ']'
++ ls /opt/kafka/libs/jmx_prometheus_javaagent-0.14.0.jar
+ KAFKA_OPTS=' -javaagent:/opt/kafka/libs/kafka-agent.jar=/var/opt/kafka/kafka-ready:/var/opt/kafka/zk-connected -Dcom.sun.management.jmxremote.port=9999 -Dcom.sun.management.jmxremote.rmi.port=9999 -Dcom.sun.management.jmxremote=true -Djava.rmi.server.hostname=240.0.244.183 -Djava.net.preferIPv4Stack=true -Dcom.sun.management.jmxremote.authenticate=false -javaagent:/opt/kafka/libs/jmx_prometheus_javaagent-0.14.0.jar=9404:/opt/kafka/custom-config/metrics-config.yml'
+ export KAFKA_OPTS
+ export LOG_DIR=/opt/kafka
+ LOG_DIR=/opt/kafka
++ tr -dc _A-Z-a-z-0-9
++ head -c32
+ CERTS_STORE_PASSWORD=47ah1jdvjnXzrtT0VXPtUjhSeI9NUWaY
+ export CERTS_STORE_PASSWORD
+ mkdir -p /tmp/kafka
+ ./kafka_tls_prepare_certificates.sh
Preparing truststore for replication listener
Adding /opt/kafka/cluster-ca-certs/ca.crt to truststore /tmp/kafka/cluster.truststore.p12 with alias ca
Certificate was added to keystore
Preparing truststore for replication listener is complete
Looking for the right CA
Found the right CA: /opt/kafka/cluster-ca-certs/ca.crt
Preparing keystore for replication and clienttls listener
Preparing keystore for replication and clienttls listener is complete
Preparing truststore for client authentication
Adding /opt/kafka/client-ca-certs/ca.crt to truststore /tmp/kafka/clients.truststore.p12 with alias ca
Certificate was added to keystore
Preparing truststore for client authentication is complete
+ echo 'Starting Kafka with configuration:'
Starting Kafka with configuration:
+ ./kafka_config_generator.sh
+ sed -e 's/sasl.jaas.config=.*/sasl.jaas.config=[hidden]/g' -e 's/password=.*/password=[hidden]/g'
+ tee /tmp/strimzi.properties
cat: /opt/kafka/custom-config/advertised-hostnames.config: No such file or directory

+ echo ''
+ '[' -z '-Xms8192m -Xmx8192m' ']'
+ . ./set_kafka_gc_options.sh
++ set -e
++ '[' true == true ']'
+++ get_gc_opts
+++ echo '-Xlog:gc*:stdout:time'
++ KAFKA_GC_LOG_OPTS='-Xlog:gc*:stdout:time'
++ export KAFKA_GC_LOG_OPTS
+ exec /usr/bin/tini -w -e 143 -- /opt/kafka/bin/kafka-server-start.sh /tmp/strimzi.properties
[2021-01-08T04:37:10.694+0000] Heap region size: 4M
[2021-01-08T04:37:10.794+0000] Using G1
[2021-01-08T04:37:10.794+0000] Heap address: 0x0000000600000000, size: 8192 MB, Compressed Oops mode: Zero based, Oop shift amount: 3
[2021-01-08T04:37:10.796+0000] Mark closed archive regions in map: [0x00000007ffc00000, 0x00000007ffc6aff8]
[2021-01-08T04:37:10.796+0000] Mark open archive regions in map: [0x00000007ffb00000, 0x00000007ffb46ff8]
2021-01-08 04:37:11,244 INFO Starting KafkaAgent with brokerReadyFile=/var/opt/kafka/kafka-ready and sessionConnectedFile=/var/opt/kafka/zk-connected (io.strimzi.kafka.agent.KafkaAgent) [main]
2021-01-08 04:37:11,245 INFO Starting metrics registry (io.strimzi.kafka.agent.KafkaAgent) [main]
2021-01-08 04:37:11,245 INFO KafkaYammerMetrics not found. Metrics will be used. (io.strimzi.kafka.agent.KafkaAgent) [main]
2021-01-08 04:37:11,710 INFO Registered kafka:type=kafka.Log4jController MBean (kafka.utils.Log4jControllerRegistration$) [main]
2021-01-08 04:37:12,345 ERROR Exiting Kafka due to fatal exception (kafka.Kafka$) [main]
org.apache.kafka.common.config.ConfigException: Missing required configuration "zookeeper.connect" which has no default value.
        at org.apache.kafka.common.config.ConfigDef.parseValue(ConfigDef.java:477)
        at org.apache.kafka.common.config.ConfigDef.parse(ConfigDef.java:467)
        at org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:108)
        at org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:142)
        at kafka.server.KafkaConfig.<init>(KafkaConfig.scala:1235)
        at kafka.server.KafkaConfig.<init>(KafkaConfig.scala:1238)
        at kafka.server.KafkaConfig$.fromProps(KafkaConfig.scala:1218)
        at kafka.server.KafkaServerStartable$.fromProps(KafkaServerStartable.scala:34)
        at kafka.server.KafkaServerStartable$.fromProps(KafkaServerStartable.scala:29)
        at kafka.Kafka$.main(Kafka.scala:68)
        at kafka.Kafka.main(Kafka.scala)
[2021-01-08T04:37:12.353+0000] Heap
[2021-01-08T04:37:12.353+0000]  garbage-first heap   total 8388608K, used 65224K [0x0000000600000000, 0x0000000800000000)
[2021-01-08T04:37:12.353+0000]   region size 4096K, 16 young (65536K), 0 survivors (0K)
[2021-01-08T04:37:12.353+0000]  Metaspace       used 13984K, capacity 14515K, committed 14720K, reserved 1062912K
[2021-01-08T04:37:12.353+0000]   class space    used 1930K, capacity 2138K, committed 2176K, reserved 1048576K

Please suggest.

[scholzj]

Did you let the operator roll all the pods between the operator upgrade and Kafka upgrade? It is not clear if you did and what was the state of that from your description. Also, as I said before ... keep in mind that the supported and tested process is to upgrade from Strimzi 0.17 to Strimzi 0.18 and then to Strimzi 0.19.

[vperi1730]

Did you let the operator roll all the pods between the operator upgrade and Kafka upgrade? - Any specific configuration inside the deployment manifest we need to take care of, I am not sure if I understood your question.

2. I just tried an upgrade from 17 to 18, I am seeing the same issue. If you see below, I have just launched 18 operator deployments and updated the Kafka version to 2.5.0.

pod/my-cluster-kafka-0                           1/2     CrashLoopBackOff   4          3m16s
pod/my-cluster-kafka-1                           2/2     Running            0          63m
pod/my-cluster-kafka-2                           2/2     Running            0          63m
pod/my-cluster-kafka-exporter-5f9f6869d4-8rrhl   1/1     Running            0          62m
pod/my-cluster-zookeeper-0                       1/1     Running            1          6m19s
pod/my-cluster-zookeeper-1                       1/1     Running            0          5m23s
pod/my-cluster-zookeeper-2                       1/1     Running            0          4m46s
pod/strimzi-operator-7f4f86b896-zf6bz            1/1     Running            0          17m

Same logs are observed.

[root@lvndev012336 cluster-operator]# kubectl logs -f my-cluster-kafka-0 -n kup20 tls-sidecar
Starting Stunnel with configuration:
pid = /usr/local/var/run/stunnel.pid
foreground = yes
debug = notice
[zookeeper-2181]
client = yes
CAfile = /tmp/cluster-ca.crt
cert = /etc/tls-sidecar/kafka-brokers/my-cluster-kafka-0.crt
key = /etc/tls-sidecar/kafka-brokers/my-cluster-kafka-0.key
accept = 127.0.0.1:2181
connect = my-cluster-zookeeper-client:2181
delay = yes
verify = 2


2021.01.08 13:11:41 LOG5[1:139977743706176]: stunnel 4.56 on x86_64-redhat-linux-gnu platform
2021.01.08 13:11:41 LOG5[1:139977743706176]: Compiled/running with OpenSSL 1.0.1e-fips 11 Feb 2013
2021.01.08 13:11:41 LOG5[1:139977743706176]: Threading:PTHREAD Sockets:POLL,IPv6 SSL:ENGINE,OCSP,FIPS Auth:LIBWRAP
2021.01.08 13:11:41 LOG5[1:139977743706176]: Reading configuration from file /tmp/stunnel.conf
2021.01.08 13:11:41 LOG5[1:139977743706176]: FIPS mode is enabled
2021.01.08 13:11:41 LOG4[1:139977743706176]: Insecure file permissions on /etc/tls-sidecar/kafka-brokers/my-cluster-kafka-0.key
2021.01.08 13:11:41 LOG5[1:139977743706176]: Configuration successful

This is for Kafka container inside my-cluster-kafka-0 pod

[2021-01-08T13:13:24.004+0000] Heap region size: 4M
[2021-01-08T13:13:24.044+0000] Using G1
[2021-01-08T13:13:24.044+0000] Heap address: 0x0000000600000000, size: 8192 MB, Compressed Oops mode: Zero based, Oop shift amount: 3
[2021-01-08T13:13:24.044+0000] Mark closed archive regions in map: [0x00000007ffc00000, 0x00000007ffc6aff8]
[2021-01-08T13:13:24.044+0000] Mark open archive regions in map: [0x00000007ffb00000, 0x00000007ffb46ff8]
2021-01-08 13:13:24,353 INFO Starting KafkaAgent with brokerReadyFile=/var/opt/kafka/kafka-ready and sessionConnectedFile=/var/opt/kafka/zk-connected (io.strimzi.kafka.agent.KafkaAgent) [main]
2021-01-08 13:13:24,353 INFO Starting metrics registry (io.strimzi.kafka.agent.KafkaAgent) [main]
2021-01-08 13:13:24,353 INFO KafkaYammerMetrics not found. Metrics will be used. (io.strimzi.kafka.agent.KafkaAgent) [main]
2021-01-08 13:13:24,685 INFO Registered kafka:type=kafka.Log4jController MBean (kafka.utils.Log4jControllerRegistration$) [main]
2021-01-08 13:13:25,206 ERROR Exiting Kafka due to fatal exception (kafka.Kafka$) [main]
org.apache.kafka.common.config.ConfigException: Missing required configuration "zookeeper.connect" which has no default value.
        at org.apache.kafka.common.config.ConfigDef.parseValue(ConfigDef.java:477)
        at org.apache.kafka.common.config.ConfigDef.parse(ConfigDef.java:467)
        at org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:108)
        at org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:142)
        at kafka.server.KafkaConfig.<init>(KafkaConfig.scala:1235)
        at kafka.server.KafkaConfig.<init>(KafkaConfig.scala:1238)
        at kafka.server.KafkaConfig$.fromProps(KafkaConfig.scala:1218)
        at kafka.server.KafkaServerStartable$.fromProps(KafkaServerStartable.scala:34)
        at kafka.server.KafkaServerStartable$.fromProps(KafkaServerStartable.scala:29)
        at kafka.Kafka$.main(Kafka.scala:68)
        at kafka.Kafka.main(Kafka.scala)
[2021-01-08T13:13:25.212+0000] Heap
[2021-01-08T13:13:25.212+0000]  garbage-first heap   total 8388608K, used 77512K [0x0000000600000000, 0x0000000800000000)
[2021-01-08T13:13:25.212+0000]   region size 4096K, 19 young (77824K), 0 survivors (0K)
[2021-01-08T13:13:25.212+0000]  Metaspace       used 15618K, capacity 16114K, committed 16256K, reserved 1062912K
[2021-01-08T13:13:25.212+0000]   class space    used 2125K, capacity 2323K, committed 2432K, reserved 1048576K

[scholzj]

Ok ... so can you describe the stapes you do exactly one by one to try to reproduce it including the YAMLs etc.? The error here suggests there is an environment variable missing here but I'm not sure why would that be. Also, are you using some of your private container images? Or are you using the Strimzi ones from Docker Hub? Are you sure the right images are used? Because for example this error looks like some version mix-up ... the Zookeeper image is expecting the environment variable. But the operator is too new and does not pass the number of Zookeeper nodes through environment variable.

[tomncooper]

What do the Zookeeper logs look like. There was a ZK upgrade from 0.17 to 0.18 (IIRC) and this required custom code to deal with changes to the ZK snapshots.

[vperi1730]

Operator_18.txt

Steps followed.

1) Installed 17 version CRD's/ClusterRoles/ClusterRoleBindings.
2) Launched the 17 operator
3) Launched Kafka CR with 2.4.0 as the app version. (Till here everything was running well)
4) Now I repeated Step(1) for the 18 version.
5) Launched the 18 operator
6) Updated the Kafka CR to version 2.5.0.
7) This has first updated all the ZK pods to 1/1(As the TLS sidecar has been removed) which was previously 2/2. After updating ZK pods, it started terminating Kafka-0 pod and during re-creation, the above error was observed.

I am using my private images which are pulled from Strimzi docker hub and then tagged & pushed into my google registry. All the versions of the operator/applications/Kafka-bridge/jmxtrans are the recommended ones I have used.

I am attaching the 18 operator file used. gcr-path in the file is my private folder path inside the
Operator_18.txt
google container.

[scholzj]

I think the YAML file is not right.

You have the operator as gcr.io/<gcr-path>/strimzi-operator:0.18.0. But the other operators are in the environment variables configuring the images you use gcr.io/<gcr-path>/strimzi-operator:0.19.0. So that seems wrong (although that would not cause the Zookeeper or Kafka issues).

Another problem is that the kafka images do not include the Strimzi version. For example gcr.io/<gcr-path>/strimzi-kafka:2.4.0. The original images always include it ... e.g. kafka:0.18.0-kafka-2.4.0. The reason for it is that the Kafka 2.4.0 image for Strimzi 0.17.0 is different from the Kafka 2.4.0 image for Strimzi 0.18.0. SO how do you distinguish these images? Because if you use exactly the same gcr.io/<gcr-path>/strimzi-kafka:2.4.0 for both Strimzi 0.17 and 0.18, this would cause exactly the issue as you had in the error in the earlier comment.

[vperi1730]

Wow, this looks to be a good catch. Then if that is the case I believe my earlier upgrade from 17->19 is also because of the same Kafka app image.

Let me check my images then which I have pulled and pushed into GCR. I will revert back on this. Thanks for today.

[scholzj]

Yeah, that would be the same.