StrimziPodSets: strimzi.io/revision differs across Kafka pods but reconciliation shows no changes

[frankgrimes97]

Hi, we've recently deployed strimzi-0.38.0/kafka-3.6.0 and have noticed the following and are unsure if it's expected...

When viewing the Kafka pods we see a different strimzi.io/revision annotation value across each node, whereas previously with strimzi-0.28.0/kafka-3.0.0 each pod managed by the StatefulSet would have the exact same controller-revision-hash label values for the pods managed by the StatefulSet.

Is it normal to expect that the new strimzi.io/revision annotation won't be consistent across the pods?
The strimzi-cluster-operator pod logs seem to show no changes to be reconciled. e.g.

2024-03-12 16:52:11 INFO  StrimziPodSetController:378 - Reconciliation #3425(watch) StrimziPodSet(<redacted>/<redacted>): StrimziPodSet will be reconciled
2024-03-12 16:52:11 INFO  StrimziPodSetController:414 - Reconciliation #3425(watch) StrimziPodSet(<redacted>/<redacted>): reconciled
…
2024-03-12 17:24:17 INFO  AbstractOperator:537 - Reconciliation #3525(timer) Kafka(<redacted>/<redacted>): reconciled

Also, the cluster appears to be running normally despite the different strimzi.io/revision annotation values.

From the docs at https://github.com/strimzi/proposals/blob/main/031-statefulset-removal.md it isn't entirely clear:

If the generation annotations differ, it decides that the pod needs to be rolled.

Is the "generation annotation" the strimzi.io/revision annotation value, or is it something else? (we suspect not since we're not seeing any rolling restarts)

[scholzj]

That is correct. The revision of the pod is based on its YAML definition. It is used to detect if something changed in it and a rolling update might be needed (as Pods are in general immutable). As each of the pods differs (different name, different volumes are mounted etc.), they have also a different revision each.

StatefulSet works a bit differently - as far as know, it bases the hash on the template which is identical for all pods as it doesn't have the differences yet rendered in it (as it is just a template).