Adding per pod anti-affinity to cluster with rack previously defined fails

[gamer22026]

Describe the bug
Adding a per pod anti-affinity stanza to a Kafka cluster spec that already has a rack property defined results in a bad affinity stanza, leading to the kafka brokers not restarting during a rolling restart

To Reproduce
Steps to reproduce the behavior:

1. Create kafka cluster with a rack property defined. This will create an preferredDuringSchedulingIgnoredDuringExecution anti-affinity section in the statefulset.
2. Once cluster if fully up, modify it by adding a per pod anti-affinity rule as outlined here: https://strimzi.io/docs/latest/#configuring-pod-anti-affinity-in-kafka-components-deployment-configuration-kafka-scheduling-based-on-pods
3. Look at the resultant kafka stateful set and see that the affinity section is incorrect:

      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - podAffinityTerm:
              labelSelector:
                matchLabels:
                  strimzi.io/cluster: kafka-cluster
                  strimzi.io/name: kafka-name
              topologyKey: failure-domain.beta.kubernetes.io/zone
            weight: 100
          requiredDuringSchedulingIgnoredDuringExecution:
          - topologyKey: ""

4. Now comment out the rack section and re-apply. Notice the affinity section in the statefulset is now correct for per pod anti-affinity.

     affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: strimzi.io/name
                operator: In
                values:
                - kafka-name
            topologyKey: kubernetes.io/hostname

5. Once cluster is fully up, re-enable the rack settings and re-apply. Notice the affinity section in the stateful set is now correct for both rack and per pod anti-affinity:

     affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - podAffinityTerm:
              labelSelector:
                matchLabels:
                  strimzi.io/cluster: kafka-cluster
                  strimzi.io/name: kafka-name
              topologyKey: failure-domain.beta.kubernetes.io/zone
            weight: 100
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: strimzi.io/name
                operator: In
                values:
                - kafka-name
            topologyKey: kubernetes.io/hostname

Expected behavior
Adding a per pod anti-affinity to an existing kafka cluster where a rack was already defined should correctly add the additional settings.

Environment (please complete the following information):

• Strimzi version: 0.15.0
• Installation method: Helm chart for operator, yaml for kafka cluster
• Kubernetes cluster: Kubernetes 1.11
• Infrastructure: AWS

[scholzj]

I tried to reproduce it on Minishift 3.11 (Based on Kube 1.11) and on Minikube with Kube 1.11.10. But I failed to reproduce it. It worked fine for me and the resulting affinity on the pods / StatefulSet was fine. From what you are describing it sounds like some issue with the patching process in Kubernetes, but without being able to reproduce it, I have no clue if we can somehow fix it.

[gamer22026]

Odd. I was able to reproduce on multiple Kafka clusters. We are running on AWS (via kops) across multiple AZ's. Not familiar with how minishift or minikube deal with implementing a simulated multiple AZ scenario. Not hard to work around in any event. I only discovered it because I had defined racks when I initially setup the clusters but had forgotten to add per pod anti-affinity.

[scholzj]

I do not think the implementation of AZs really matters ... the affinity here really seems to fail at the YAML level not at the implementation level. I'm going to try it on my AWS cluster as well, but that is using newer version of Kubernetes, so I'm not sure how relevant that is.

[scholzj]

FYI: I gave it a try, but that also didn't show any issues. Sorry.

[scholzj]

I will close this since there was no update for more than 10 days. If you have something more feel free to open a new issue, reopen this one or get in touch with us on Slack or mailing list.