Only first pod of StatefulSet updated on modification of SearchHeadCluster or IndexerCluster (e.g. image, resources)

[romain-bellanger]

Hello,

This is an impact of #193: on change of SearchHeadCluster or IndexerCluster custom resource properties such as image or resource quotas, only the first pod of the associated StatefulSet is updated. The purpose of this PR was precisely to update the controller-revision-hash label of all the pods to prevent their restart.

We are currently working on the upgrade of clusters in Splunk 8.0.6 deployed with the operator alpha version 0.1.0, to Splunk 8.1.0.1 with the operator beta version 0.2.1. This breaks a lot of automation in the upgrade process. When changing the docker image, pods were previously restarted sequentially with a proper offline (for indexers), which was matching the requirements for a clean upgrade (with just a couple of additional API calls to initialize / finalize the process). With this change only the first pod is restarted automatically. The upgrade of the remaining pods must be done by running offline from inside the container and deleting the pod manually.

The same impact probably exists on all the specs from custom resources inherited by the pods (I tested for instance modifying the CPU limit, again only the first pod was updated).

For now, we patched the operator by commenting the first call to checkAndUpdatePodRevision from method UpdateStatefulSetPods (case of readyReplicas < replicas), which fixed the problem. But we don't understand the impact of this modification, as we don't understand the initial purpose of the PR. We never observed cases of unnecessary pod recycle on scaling. Could you please describe these scenarios, and look for a way to address them without causing such an impact on change of CRD specs?

[gaurav-splunk]

Hi @romain-bellanger Thanks for finding the issue. Let me check it at my end and I'll let you know about the findings and fix at my end.

But we don't understand the impact of this modification, as we don't understand the initial purpose of the PR. We never observed cases of unnecessary pod recycle on scaling.

We observed in our environment that whenever we tried to scale up/down indexer cluster, it recycled the indexer pods since the controller-revision-hash has changed and in our code, we were recycling the pods if there is a change in that label. I am not sure how you tried scaling. We tried using kubectl scale idxc <idxc-name> --replicas=<new replica count>.

[gaurav-splunk]

Hi @romain-bellanger , hopefully this PR should fix the issue you are seeing.

[romain-bellanger]

Hi @gaurav-splunk, thanks a lot for checking this so quickly!
We mostly experimented auto-scaling, which doesn't modify the controller-revision-hash. Then we are generally using Openshift templates to describe our custom resources, the number of replicas being a parameter of the template. But we haven't used this for a while to scale clusters, and I didn't remember about these unexpected recycles. But this operation changes controller-revision-hash so the impact was most probably there.

[akondur]

Hi @romain-bellanger , this issue should be fixed via the following PR. Could you please check with the latest release?

[akondur]

Closing as the issue is fixed in develop/latest release.