Optimization of pod batch restart in cluster upgrade #667
Conversation
…er freezing forbidden error when updating statefulset. codes will not enter the subsequent process of deleting and reconstructing statefulset. Pods are no longer restarted when new shards/replicas are added, modified and deleted.
|
@alex-zaitsev could you take a look at this pr please? |
|
@sunsingerus Excuse me, is there any problem with this pr, or is there any better solution to this problem? |
| @@ -253,7 +253,7 @@ func (w *worker) updateCHI(old, new *chop.ClickHouseInstallation) error { | |||
| WithStatusAction(new). | |||
| M(new).F(). | |||
| Info("reconcile started") | |||
| w.a.V(2).M(new).F().Info("action plan\n%s\n", actionPlan.String()) | |||
| //w.a.V(2).M(new).F().Info("action plan\n%s\n", actionPlan.String()) | |||
There was a problem hiding this comment.
Was it intentional to disable this log line? (If so: why not remove it completely?)
There was a problem hiding this comment.
When debugging, this will report null pointer exception. I wonder if it is an environment problem, so do this.
|
I tried this patchset / PR in our cluster, since I had a similar problems (pods being restarted completely when labels change), but now they don't update at all. If I change the clickhouse version in a podtemplate and apply the new custom resource definition, no new changes are rolled out. The labels are updated, but that's it :-/ |
PR is incorrect. In order to solve the original issue we will remove those confusing labels at all. They were more for debugging purposes. |
This PR is to solve the problem of pod restart. The pod tag has forgotten to update in the code. You can be updated without my PR, which will inevitably lead to the restart of pod. |
Now, each sts only controls one pod. I don't think there is a need to configure tags on pods. Just configure tags on sts. I don't know what's wrong with my PR. |
Optimize the statefulset update, patching labels first will not trigger forbidden error when updating statefulset.
codes will not enter the subsequent process of deleting and reconstructing statefulset.
Pods are no longer restarted when new shards/replicas are added, modified and deleted.